Systems and methods for performing data replication

ABSTRACT

Preparing source data to be replicated in a continuous data replication environment. Certain systems and methods populate a file name database with entries having a unique file identifier descriptor (FID), short name and a FID of the parent directory of each directory or file on a source storage device. Such information is advantageously gathered during scanning of a live file system without requiring a snapshot of the source storage device. The database can be further used to generate absolute file names associated with data operations to be replayed on a destination storage device. Based on the obtained FIDs, certain embodiments can further combine write operations to be replayed on the destination storage device and/or avoid replicating temporary files to the destination system.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.13/118,294, filed on May 27, 2011, U.S. Pat. No. 8,589,347, which claimsthe benefit of priority under 35 U.S.C. §119(e) of U.S. ProvisionalPatent Application No. 61/349,624, filed on May 28, 2010, and entitled“SYSTEMS AND METHODS FOR PERFORMING DATA REPLICATION,” the disclosuresof which are hereby incorporated by reference in their entireties.

BACKGROUND

1. Field

The present disclosure relates to performing copy and/or data managementoperations in a computer network and, in particular, to systems andmethods for performing data replication in a storage management system.

2. Description of the Related Art

Computers have become an integral part of business operations such thatmany banks, insurance companies, brokerage firms, financial serviceproviders, and a variety of other businesses rely on computer networksto store, manipulate, and display information that is constantly subjectto change. Oftentimes, the success or failure of an importanttransaction may turn on the availability of information that is bothaccurate and current. Accordingly, businesses worldwide recognize thecommercial value of their data and seek reliable, cost-effective ways toprotect the information stored on their computer networks.

To address the need to maintain current copies of electronicinformation, certain data replication systems have been provided to“continuously” copy data from one or more source machines to one or moredestination machines. These continuous data replication (CDR) systemsprovide several advantages for disaster recovery solutions and cansubstantially reduce the amount of data that is lost during anunanticipated system failure.

One drawback of such CDR systems is that, during an initialsynchronization phase, many systems record absolute file names whenscanning a source storage device in order to replicate the scanned datato a same location on a destination storage device. Moreover, thisscanning is generally performed while the source file system is in afixed state, such as based on a snapshot of the file system.

SUMMARY

In view of the foregoing, a need exists for improved systems and methodsfor preparing and transmitting source data to be replicated to adestination system. For instance, there is a need for systems andmethods for scanning a live file system during an initialsynchronization phase between the source and destination systems.

Certain embodiments of the invention are provided for intelligent datareplication. In particular, embodiments of the invention includeimproved systems and methods scanning a source file system having datato be copied in a CDR system. In certain embodiments, such scanning isperformed on a live file system without requiring a snapshot of the filesystem data. For example, file identifier descriptors (FIDs), which aregenerally not exposed to outside the kernel, of files and/or directorieson the file system can be used to populate a file name database usableto construct an absolute file name when transmitting data to areplication system.

In certain further embodiments, use of FIDs to track files on the sourcesystem and/or an introduction of a slight delay in the replicationprocess allows for more efficient data replication operations. Forinstance, journal entries representing monitored data operations on thesource file system can be stored without storing the actual data.Moreover, location information in the log entries can be analyzed todetermine if multiple write operations on the source system can becombined into a single write operation on the destination system. In yetother embodiments, temporary files on the source system can beidentified and not copied to the destination system.

According to certain embodiments, a method is provided for identifyingdata to be copied in a data replication system. The method can includeobtaining with a scanning module executing on a computing device a firstfile identifier descriptor (FID) of a first directory on a live sourcefile system. In some cases, the first FID is one of a plurality ofunique identifiers corresponding to a plurality of directories and fileson the source file system. The method may further include adding thefirst FID to a queue, and can also include storing a current journalsequence number from a file system filter driver identifying a firsttime. In some instances, the method includes, following said storing,accessing a current directory of the plurality of directories on thesource file system that corresponds to a next FID stored in the queue.The method can additional include obtaining additional FIDs for eachimmediate child directory and immediate child file in the currentdirectory. If no changes have been made to the current directory sincethe first time, the method can include: populating a file name databasewith the additional FIDs of each immediate child directory and immediatechild file in the current directory; adding the additional FIDs of eachimmediate child directory of the current directory to the queue; and/orremoving the next FID from the queue. If changes have been made to thefirst directory since the first time, the method can include repeatingsaid storing, said accessing and said obtaining the additional FIDs.

In some embodiments, a system is provided for preparing data forreplication from a source computing device in a network. The may includea queue configured to store a plurality of file identifier descriptors(FIDs) each comprising a unique identifier that corresponds to one of aplurality of directories and files on a source file system. The systemcan also include a scanning module executing on a computing device andconfigured to scan the source file system while in a live state and topopulate the queue with the plurality of FIDs. In certain cases, thesystem additionally includes a database comprising file name data thatassociates each of the plurality of FIDs with a short name and a parentFID. The scanning module can be further configured to populate thedatabase with the file name data based on said scan of the source filesystem in the live state. The system can also include at least onedatabase thread configured to receive a data entry identifying a datamanagement operation associated with at least one of the plurality ofdirectories and files on the source file system and to construct fromthe FID associated with the at least one directory or file an absolutefile name for transmission to a destination system along with a copy ofthe data management operation for replying on the destination system.

According to other aspects of the disclosure, a method is provided forperforming data replication. The method can include monitoring aplurality of journal entries associated with writing data to a sourcestorage device. The method may further include identifying a firstjournal entry of the plurality of journal entries. The first journalentry may comprise a first data write operation, a first file identifierdescriptor (FID) of a file to be modified by the first data writeoperation on the source storage device, and a first location of a firstportion of the file to be modified. The method can also includeidentifying a second journal entry of the plurality of journal entries,the second journal entry comprising a second data write operation, asecond FID of a file to be modified by the second data write operationon the source storage device, and a second location of a second portionof the file to be modified. In some instances, the method additionallyincludes determining that the first and second data write operations canbe combined into a single write operation. The method may also includeconstructing an absolute file name based on at least one of said firstand second FIDs, wherein neither the first nor second journal entriescomprises the absolute file name. In some embodiments, the methodincludes transmitting the single write operation and the absolute filename to a destination storage device to replay on the destinationstorage device the data modifications associated with the first andsecond write operations.

According to yet further aspects of the disclosure, a system is providedfor performing data replication. The system can include at least onecomputer application executing on a computing device and configured togenerate operations associated with data on a source storage device. Thesystem may also include a filter module disposed between the at leastone computer application and the first storage device. The filter modulecan be configured to identify from the operations, a first datamodification operation, a first file identifier descriptor (FID) of afile to be modified by the first data modification operation, and afirst location of a first portion of the file to be modified, and asecond data modification operation, a second FID of a file to bemodified by the second data modification operation, and a secondlocation of a second portion of the file to be modified. The system canfurther include a processing module configured to determine that thefirst and second data modification operations can be combined into asingle modification operation. In some embodiments, the system alsoincludes at least one database thread configured to construct anabsolute file name for replaying the single modification operation onreplication data of a destination storage device based on at least oneof said first and second FIDs. In some cases, neither the first norsecond data modification operations comprises the absolute file name.

According to other embodiments, a system is provided for performing datareplication. The system can include means for monitoring a plurality ofjournal entries associated with writing data to a source storage device.The system can further include means for identifying a first journalentry of the plurality of journal entries, the first journal entrycomprising a first data write operation, a first file identifierdescriptor (FID) of a file to be modified on the source storage device,and a first location of a first portion of the file to be modified, andfor identifying a second journal entry of the plurality of journalentries, the second journal entry comprising a second data writeoperation, a second FID of a file to be modified on the source storagedevice, and a second location of a second portion of the file to bemodified. The system can also include means for determining that thefirst and second data write operations can be combined into a singlewrite operation. In certain embodiments, the system further includesmeans for constructing an absolute file name based on at least one ofsaid first and second FIDs, wherein neither the first nor second journalentries comprises the absolute file name. The system may additionallyinclude means for transmitting the single write operation and theabsolute file name to a destination storage device to replay on thedestination storage device the data modifications associated with thefirst and second write operations.

According to additional aspects, a method is provided for performingdata replication. The method can include monitoring data operationsassociated with an application executing on a computing device, the dataoperations operative to write data to a first storage device. The methodcan also include populating a log file with a plurality of data entriesindicative of the data operations. In some cases, the method includesidentifying a first one of the plurality of data entries associated withwriting data to a temporary file on the first storage device. The methodmay additionally include replaying to a second storage device, based ona portion of the data entries, a portion of the data operations toreplicate data to a first location on the second storage device. Theportion of the data entries according to some embodiments does notinclude the first one of the plurality of data entries.

According to some aspects of the disclosure, a system is provided forperforming data replication between two computing devices in a network.The system can include at least one computer application executing on afirst computing device and configured to generate a plurality ofoperations associated with storing data on a source storage device, thedata comprising at least one temporary file and at least onenon-transitory file. The system may also include a log file comprising aplurality of data entries indicative of the plurality of operations. Insome cases, a first one of the plurality of data entries is associatedwith writing the at least one temporary file. According to someembodiments, the system includes a processing module executing on andconfigured to identify a first one of the plurality of data entriesassociated with writing the temporary file. The processing module may befurther configured to copy a portion of the entries of the log file to asecond computing device in network communication with the firstcomputing device. The portion of the data entries according to someembodiments does not include the first one of the plurality of dataentries.

For purposes of summarizing the disclosure, certain aspects, advantagesand novel features of the inventions have been described herein. It isto be understood that not necessarily all such advantages may beachieved in accordance with any particular embodiment of the invention.Thus, the invention may be embodied or carried out in a manner thatachieves or optimizes one advantage or group of advantages as taughtherein without necessarily achieving other advantages as may be taughtor suggested herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of a data replication systemaccording to certain embodiments of the invention.

FIG. 2 illustrates a block diagram of an exemplary embodiment of asource system of the data replication system of FIG. 1.

FIG. 3 illustrates a block diagram of an exemplary embodiment of adestination system of the data replication system of FIG. 1.

FIG. 4 illustrates a block diagram of further details of an exemplaryembodiment of the data replication system of FIG. 1.

FIG. 5 illustrates various fields of an exemplary embodiment of a logentry usable by the data replication systems of FIGS. 1 and 4.

FIG. 6 illustrates a block diagram of an exemplary embodiment of thedata replication system of FIG. 1 having a storage manager module.

FIG. 7 illustrates a flow chart of an exemplary embodiment of aninstallation process usable by the data replication system of FIG. 4.

FIG. 8 illustrates a flow chart of an embodiment of a process of takinga consistency replication point usable by the data replication system ofFIG. 4.

FIG. 9 illustrates a block diagram of an exemplary embodiment of apathname translation system usable with embodiments of a datareplication system.

FIG. 10 illustrates an exemplary embodiment of a pathname translationdatabase usable with embodiments of the pathname translation system ofFIG. 9.

FIG. 11 illustrates a flowchart of an exemplary embodiment of a pathnametranslation process executable by the pathname translation system ofFIG. 9.

FIG. 12 illustrates a block diagram of another embodiment of a pathnameor file name translation system usable with embodiments of a datareplication system.

FIG. 13 illustrates a flowchart of a process for scanning a live filesystem, according to certain embodiments of the invention.

FIG. 14 illustrates a flowchart of a scan routine usable within theprocess of FIG. 13, according to certain embodiments of the invention.

FIGS. 15A and 15B illustrate a flowchart of a replication process forinterleaving a stream of journal entries with database results of a livefile system scan, according to certain embodiments of the invention.

FIG. 16 illustrates a flowchart of another replication process usablewith embodiments of a data replication system.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

As will be seen from the disclosure herein, certain embodiments ofsystems and methods are provided for intelligent data replication. Inparticular, embodiments of the invention include improved systems andmethods for scanning a source file system having data to be copied in aCDR system. In certain embodiments, such scanning is performed on a livefile system without requiring a snapshot of the file system data. Forexample, FIDs of files and/or directories on the file system can be usedto populate a file name database usable to construct an absolute filename when transmitting data to the replication system.

In certain further embodiments, use of FIDs to track files on the sourcesystem and/or accumulating a group of journal entries to transmit duringthe replication process allows for more efficient data replicationoperations. For instance, logs with entries representing monitored dataoperations on the source file system can be stored without actual data.Moreover, location information in the log entries can be analyzed todetermine if multiple write operations on the source system can becombined into a single write operation on the destination system. In yetother embodiments, temporary files on the source system can be passedover when identifying source data to be copied to the destinationsystem.

The features of the systems and methods will now be described withreference to the drawings summarized above. Throughout the drawings,reference numbers are re-used to indicate correspondence betweenreferenced elements. The drawings, associated descriptions, and specificimplementation are provided to illustrate embodiments of the inventionand not to limit the scope of the disclosure.

Moreover, embodiments of the invention can be used in combination withreplication systems and methods described in U.S. Pat. No. 7,651,593,which is hereby incorporated herein in its entirety to be consideredpart of this specification.

FIG. 1 illustrates a block diagram of a data replication system 100according to certain embodiments of the invention. As shown, thereplication system 100 comprises a source system 102 capable ofcommunicating with a destination system 104 by sending and/or receivingdata over a network 106. For instance, in certain embodiments, thedestination system 104 receives and/or stores a replicated copy of atleast a portion of data, such as application-specific data, associatedwith the source system 102.

The illustrated network 106 advantageously comprises any means forcommunicating data between two or more systems or components. It certainembodiments, the network 106 comprises a computer network. For example,the network 106 may comprise a public network such as the Internet,virtual private network (VPN), token ring or TCP/IP based network, widearea network (WAN), local area network (LAN), an intranet network,point-to-point link, a wireless network, cellular network, wireless datatransmission system, two-way cable system, interactive kiosk network,satellite network, broadband network, baseband network, combinations ofthe same or the like. In embodiments wherein the source system 102 anddestination system 104 are part of the same computing device, thenetwork 106 may represent a communications socket or other suitableinternal data transfer path or mechanism.

As shown, the source system 102 comprises one or more applications 108residing on and/or being executed by a computing device. For instance,the applications 108 may comprise software applications that interactwith a user to process data and may include, for example, databaseapplications (e.g., SQL applications), word processors, spreadsheets,financial applications, management applications, e-commerceapplications, browsers, combinations of the same or the like. Forexample, in certain embodiments, the applications 108 may comprise oneor more of the following: MICROSOFT EXCHANGE, MICROSOFT SHAREPOINT,MICROSOFT SQL SERVER, ORACLE, MICROSOFT WORD and LOTUS NOTES.

The source system 102 further comprises one or more processes, such asfilter drivers 110, that interact with data (e.g., production data)associated with the applications 108. For instance, the filter driver110 may comprise a file system filter driver, an operating systemdriver, a filtering program, a data trapping program, an application, amodule of the application 108, an application programming interface(“API”), or other like software module or process that, among otherthings, monitors and/or intercepts particular application requeststargeted at a file system, another file system filter driver, a networkattached storage (“NAS”), a storage area network (“SAN”), mass storageand/or other memory or raw data. In some embodiments, the filter driver110 may reside in the I/O stack of the application 108 and mayintercept, analyze and/or copy certain data traveling from theapplication 108 to a file system.

In certain embodiments, the filter driver 110 may intercept datamodification operations that include changes, updates and newinformation (e.g., data writes) with respect to the application(s) 108of interest. For example, the filter driver 110 may locate, monitorand/or process one or more of the following with respect to a particularapplication 108, application type or group of applications: datamanagement operations (e.g., data write operations, file attributemodifications), logs or journals (e.g., NTFS change journal),configuration files, file settings, control files, other files used bythe application 108, combinations of the same or the like. In certainembodiments, such data may also be gathered from files across multiplestorage systems within the source system 102. Furthermore, the filterdriver 110 may be configured to monitor changes to particular files,such as files identified as being associated with data of theapplications 108.

In certain embodiments, multiple filter drivers 110 may be deployed on acomputing system, each filter driver being dedicated to data of aparticular application 108. In such embodiments, not all informationassociated with the client system 102 may be captured by the filterdrivers 110 and, thus, the impact on system performance may be reduced.In other embodiments, the filter driver 110 may be suitable for use withmultiple application types and/or may be adaptable or configurable foruse with multiple applications 108. For example, one or more instancesof customized or particularizing filtering programs may be instantiatedbased on application specifics or other needs or preferences.

The illustrated source system 102 further comprises a source storagedevice 112. The source storage device 112 may include any type of mediacapable of storing data. For example, the source storage device 112 maycomprise magnetic storage (such as a disk or a tape drive) or other typeof mass storage. In certain embodiments, the source storage device 112may be internal and/or external to (e.g., remote to) the computingdevice(s) having the applications 108 and the filter drivers 110.

As further illustrated in FIG. 1, the destination system 104 comprises areplication module 114 and a destination storage device 116. In certainembodiments, the replication module 114 is configured to monitor and/ormanage the copying of data from the source system 102 to the destinationsystem 104, such as data retrieved by the filter drivers 110. In yetother embodiments, the replication module 114 is a “dumb” server orterminal that receives and executes instructions from the source system102.

The destination storage device 116 may include any type of media capableof storing data, such as replication data sent from the source system102. For example, the destination storage 116 device may comprisemagnetic storage (such as a disk or a tape drive) or other type of massstorage. In certain embodiments, the destination storage device 116 maybe internal and/or external to the computing device(s) having thereplication module 114.

In certain embodiments, the source storage device 112 and/or thedestination storage device 116 may be implemented as one or more storage“volumes” that include physical storage disks defining an overalllogical arrangement of storage space. For instance, disks within aparticular volume may be organized as one or more groups of redundantarray of independent (or inexpensive) disks (RAID). In certainembodiments, either or both of the storage devices 112, 116 may includemultiple storage devices of the same or different media.

FIG. 2 illustrates a block diagram of an exemplary embodiment of thesource system 102 of FIG. 1. In particular, the source system 102comprises a client computer 230 on which the application(s) 108 and thefilter driver(s) 110 reside and/or are executed. In certain embodiments,the client computer 230 comprises any computing device capable ofprocessing data and includes, for example, a server computer, aworkstation, a personal computer, a cell phone, a portable computingdevice, a tablet computer, a handheld computing device, a personaldigital assistant (PDA) or the like.

The illustrated client computer 230 further comprises a file system 234for organizing files and directories accessible by the client computer230. In certain embodiments, the file system 234 comprises a datastructure usable to keep track of a collection of files and/ordirectories stored on the source storage device 112. The file system 234may include, for example, a local file system, a network file system, afile server, a management program or the like, or may include multiplefile systems accessible by an operating system. For instance, inembodiments wherein the storage device 112 is associated with multiplevolumes, each volume may be associated with its own file system 234, ora single file system 234 may span across the multiple volumes.

The illustrated client computer 230 also comprises one or more dataagents 236. In certain embodiments, the data agent 236 comprises amodule responsible for performing data and/or storage tasks related tothe client computer 230. For example, the data agent 236 may manageand/or coordinate the compilation of and/or transferring of replicationdata from the source system 102. In other embodiments, the data agent236 may provide archiving, migrating, and/or recovery of client computerdata.

In certain embodiments, the client computer 230 comprises a plurality ofdata agents 236, each of which performs data management operationsrelated to data associated with each application 108. In suchembodiments, the data agent 236 may be aware of the various files,folders, registry files and/or system resources that are impacted by aparticular application 108. For instance, the data agent 236 may beprogrammed to detect data management requests by a particularapplication 108 and determine which files, folders and/or systemresources are associated with the data management requests.

In certain embodiments, the data agent 236 is configured to perform datamanagement operations in accordance with one or more “storage policies”or other preferences. A storage policy may include a data structure orother information having a set of preferences and other storage criteriafor performing a storage operation. The preferences and storage criteriamay include, but are not limited to, information regarding storagelocations, relationships between system components, network pathways,retention policies, data characteristics, compression or encryptionrequirements, preferred system components, combinations of the same orthe like.

In certain embodiments, one or more data agents 236 are configured toperform an initial “seeding” or synchronization process of a replicationprocess. For example, prior to (or concurrently with) data replicationusing one or more filter drivers 110, the data agent 236 may perform ascan of the source system 102 (e.g., the source storage device 112). Forinstance, the data agent 236 may evaluate the folders and/or directorystructure of the source system 102 to determine which folders are usedby a particular application 108. In certain embodiments, the data agent236 may also identify, arrange, and queue necessary data of theapplication 108 to provide a proper platform for replication. Forexample, the data agent 236 may populate source log(s) 244 withapplication data that has already been written to the source storagedatabase 112. In certain embodiments, this populating is performed basedon a snapshot or point-in-time copy of the file system. In yet otherembodiments, as described below, the data agent 236 is configured toscan a live file system.

In certain embodiments, when the data agent 236 is initially installedor enabled on the client computer 230, the data agent 236 may evaluatethe application 108. For instance, the data agent 108 may determine theapplication's organizational structure, which may include, for example,folder, directory and file information. The information gathered by thedata agent 236 may be sufficient to define a complete “set” ofinformation to be replicated such that suitable baseline datarepresenting the current operational state of the application 108 isidentified. In some instances, this initial process may require theexamination and identification of data related to application operationsoccurring prior to the installation of data agent 236. The data agent236 may also be configured to identify general configuration andoperational information regarding the application 108.

In certain embodiments, the data agent 236 may be configured to accessand/or monitor particular files, folders, directories, registries,preferences and/or other like data structures for information to bereplicated. All or a portion of the information gathered by the dataagent 236 may be copied over to the destination system 104 as part ofthe initial seeding or initialization process. After the seeding processis complete, data replication may occur on a substantially continuousbasis based on data transfers occurring between application(s) 108 andsource storage device 112. In certain embodiments, the seeding processmay occur substantially concurrently with execution of theapplication(s) 108. For instance, data operations from theapplication(s) 108 may be temporarily stored in a queue or buffer untilthe seeding process, or a portion thereof, is complete.

As shown in FIG. 2, the client computer 230 communicates through thefile system 234 with the source storage device 112, which furtherincludes a database 240 and database logs 242. In yet other embodiments,the client computer may communicate with NAS or the like. In certainembodiments, data intended for the source storage device 112 may befirst written to a file in the database logs 242 and subsequentlycommitted to the database 240 in accordance with data managementtechniques for enhancing storage operation performance. Moreover,although only one database 240 and one database log 242 are depicted inFIG. 2, it will be understood that the source storage device 112 maycomprise additional databases 240, database logs 242 and/or otherdirectory and file storage structures to meet the storage needs of theclient computer 230.

As illustrated in FIG. 2, the filter driver 110 is advantageouslylocated between the application 108 and the file system 234. Forinstance, the filter driver 110 may be deployed in the stack as an I/Obuffer and/or process in the data path between the application 108 andthe file system 234. In such embodiments, the filter driver 110 mayintercept, snoop, supervise, trap, process or otherwise be cognizant ofsome or all operations (e.g., data modification operations, filemodification operations, read operations and the like) from theapplication 108 to its associated location(s) on the source storagedevice 112.

For example, in certain embodiments, the filter driver 110 maycommunicate with an associated data agent 236 to determine where datafor a particular application 108 will be stored (e.g., particularfolders on the file system 234). In certain embodiments, the filterdriver 110 and/or the data agent 236 may also monitor and/or parse datamanagement operations to determine if new or additional folders areaffected by the production volume data of the particular application108. In certain embodiments, the data agent 236 may monitor datamanagement operations and/or other data for other purposes, such as, forexample, for satisfying a query or command by a storage managercomponent or the like.

As further depicted in FIG. 2, one or more of the filter drivers 110 andassociated data agent(s) 236 may be grouped together as a single module,such as driver module 237. In yet other embodiments, the data agent(s)236 may be separate from the driver module 237.

As discussed above, in certain embodiments, the filter driver 110 ispreferably configured to monitor and/or filter data managementoperations associated with a particular application 108. The filterdriver 110 may be further configured, according to predefined criteria,to cause particular data to be written to one or more source logs 244for subsequent replication. For instance, the filter driver 110 may beconfigured to intercept, scrub, parse and/or trap data managementoperations and to populate the source logs 244 with changes associatedtherewith.

In certain embodiments, the filter driver 110 may examine the datamanagement operation in progress, determine whether the type ofoperation is one of interest for replication purposes, and/or copyselect or all data to source log 244. For instance, as discussed above,the filter driver 110 may determine if the data management operationconcerns data in one or more files determined as relevant to replication(e.g., files that may store data for a particular application). In otherembodiments, the filter driver 110 may generate log entries for all datamanagement operations.

The filter driver 110 may further process and/or traverse the data andcopy, generate or examine other relevant information, such as a logentry number, time information (e.g., time stamp), application type,data size and start field, combinations of the same or the like, thatmay be useful in the replication process. In other embodiments, thefilter driver 110 may monitor files on the source storage device 112 formodifications of data relating to the subject application 108. Forinstance, as disclosed above, the filter driver 110 may monitor a selectgroup of files, which have been associated with the application 108, orfolders to detect changes to data stored therein. In certainembodiments, the filter driver 110 or other system component may detectwhen a data write operation of the application is made to a file orfolder not in the select group. The filter driver 110 or other systemcomponent may then determine from the properties of the data writemodification if the subject folder or file should be added to the selectgroup (for subsequent monitoring).

In certain embodiments, the filter driver 110 is deployed (e.g., by dataagent 236) on the client computer 230 prior to the beginning of thereplication process. In embodiments wherein the filter driver 110 isdeployed after replication begins, pertinent application data alreadystored on the source storage device 112 may be copied to the source logs244 prior to the replication process (e.g., during the initial “seeding”process described above).

In certain embodiments, the filter driver 110 may be enabled and/ordisabled by the data agent 236. For instance, enabling the filter driver110 may allows it to populate an associated source log 244 with logentries from application data passed from the application 108 to thesource storage device 112. When the filter driver 110 is disabled, datamay pass directly through to the source storage device 112 without beingcopied to the source logs 244.

In certain embodiments, the data agent 236 monitors the storage capacityof the source logs 244. For instance, when one or more of the sourcelogs 244 reach a particular memory threshold, the data agent 236 mayopen a socket and communicate to the destination system 104 that a copyof the source log 244 is ready to be transmitted. In other embodiments,the data agent 236 may be configured to copy the source log 244 to thedestination system 104 at periodic intervals or in accordance with otherpredefined criteria. In yet other embodiments, the source logs maintainthe history of previous intercepted changes (e.g., the last N gigabytesof previous changes). As just one example scenario, the history ofintercepted changes can be used in the event that network connectivityis temporarily lost. For example, the history of intercepted changes canbe accessed, and any changes that were not transmitted due to theconnectivity interruption can be transmitted or retransmitted to theappropriate destination. This may be particularly useful where there aremultiple destination devices and where the changes are successfullytransmitted to a first subset of the multiple destination devices, butnot a second subset of the multiple destination devices. In this case,the history can be accessed to transmit or retransmit the appropriateintercepted changes to the second subset of destination devices.

In certain embodiments, the source system 102 communicates with theassociated destination system to verify that the two systems aresynchronized. For instance, the source system 102 may receive from thedestination system an identification (e.g., unique serial number) of thedata write operation currently being replicated by the destinationsystem. The source system 102 may then compare the receivedidentification with data write operation being forwarded to the sourcestorage device 112.

FIG. 3 illustrates a block diagram of an exemplary embodiment of thedestination system 104 of FIG. 1. In particular, the destination system104 comprises the replication module 114, which communicates with one ormore replication logs 352 and the destination storage device 116. Incertain embodiments, the replication module 114 comprises any computingdevice capable of processing data and includes, for example, a servercomputer, a workstation, a personal computer or the like.

In certain embodiments, the replication logs 352 contain a copy of thedata stored on the source logs of a client system, such as the sourcelogs 244 of FIG. 2. The replication logs 352 comprise any type of memorycapable of storing data including, for example, cache memory. In certainembodiments, the replication logs 352 may reside on the destinationsystem 104, such as, for example, on the destination storage device 116,or at least a portion of the replication logs 352 may be external to thedestination system 104. In certain embodiments, once the replicationlogs 352 have been populated with the data from the source logs 244, thedata on the source logs 244 is available to be erased and/or overwrittento conserve memory space.

The replication module 114 of the destination system 104 furthercomprises a replication agent 356 and one or more processes, such asthreads 358. In certain embodiments, the replication agent 356 comprisesone or more software modules that coordinate the transfer of data fromthe replication logs 352 to the destination storage device 116.

For example, in certain embodiments, the replication agent 356instantiates an appropriate number of threads, processes, or routines,358 for copying data from the replication logs 352 to the destinationstorage device 116. In certain embodiments, the number of threads 358 isbased on one or more of the following factors: the number of log filessent from the source logs 244 to the replication logs 352, informationreceived from the data agent(s) 236, information generated by the filterdriver(s) 110, and the type(s) of application data being tracked.

In certain embodiments, the replication agent 356 further includesmapping or correlation information that determines when and to where thedata from the replication logs 352 is copied by the threads 358. Incertain embodiments, such mapping information may be based on system- oruser-defined parameters and/or may be automatically generated, such asbased on the status of the destination storage device 116.

The one or more threads 358 (or processes) direct movement of data fromreplication logs 352 to the appropriate location on the destinationstorage device 116. In operation, in certain embodiments, the threads358 advantageously process (or traverse) replication logs 352 forparticular types of data and then copy that data to certain locations onone or more replication volumes based on data paths identified by thereplication agent 356 and/or associated with each thread 358. Forexample, the thread(s) 358 may sequentially process each entry in thereplication log 352 and write the associated data to the destinationstorage device 116.

In certain embodiments, each thread 358 is assigned to a hard-coded pathpair, which includes (i) a source path identifying the location on thesource storage device 112 associated with a data management operation(e.g., c:\Folder\) and (ii) a destination path identifying the locationon the destination storage device 116 to receive the replicated data(e.g., D:\folder\) from the thread 358.

FIG. 4 illustrates further details of a replication system 400 inaccordance with certain embodiments of the invention. As illustrated,the replication system 400 comprises the source system 102 incommunication with the destination system 104, portions of which aredescribed in more detail with respect to FIGS. 1-3.

As detailed above, in certain embodiments, the filter driver 110preferably substantially continuously populates data relating to one ormore of the applications 108 to the source logs 244. As shown in FIG. 4,the source logs 244 further comprise a first log file 460 and a secondlog file 462. In certain embodiments, the filter driver 110 sequentiallywrites log entries to the source logs 244, and when a certain capacityof the first log file 460 is reached, the filter driver 110 beginspopulating the second log file 462 with log entries.

In yet other embodiments, data relating to each application 108 ofinterest may be written to a particular log file established for thatapplication. For example, with reference to FIG. 4, the first log file460 may relate to a first application of interest, whereas the secondlog file 462 may relate to a second application of interest.

In certain embodiments, each of the log files of the source logs 244 maybe established by the data agent(s) 236 and/or the filter driver(s) 110as part of an initial deployment or initialization process. Moreover,data may be written to the source logs 244 as determined by preferencesstored on or accessed by the client computer 230 in a preferencedatabase 465.

For example, as further shown in FIG. 4, the first and second log files460, 462 may comprise a series of entries, each having an identifierthat indicates the sequence order and/or type of entry being made. Forinstance, the illustrated entry identifier (“L1”) may indicate that theparticular entry represents a first database entry in a particular orderof operation. The illustrated entry identifier (“L2”) may indicate asecond database entry in a particular order of operation, and so forth.The illustrated entry identifier (“D1”) may indicate that the particularentry represents a first database commit entry in a particular order ofoperation. Thus, in the example described above, the log entriesidentified by L1 and L2 may correspond to modifications associated witha particular database transaction, and the log entry identified by D1may correspond to a commit command for the particular transaction.

It will be understood that, although only two log files are shown inFIG. 4, more or fewer log files may be used with embodiments of theinvention. For instance, multiple applications 108 may be monitored bythe filter drivers 110 and, thus, additional log files may be added asnecessary or desired. Moreover, although in some embodiments, eachapplication 108 and each log file in the source logs 244 may have itsown associated filter driver 110, in other embodiments, a single filterdriver 110 may be deployed and configured for use with multipleapplications 108 such that there are separate log files for eachmonitored application 108.

With continued reference to FIG. 4, in certain embodiments of theinvention, the data agent 236 and/or filter driver 110 may beadvantageously configured to pause, or quiesce, the application 108during data replication. For instance, the data agent 236 may cause theapplication 108 to temporarily suspend data management operations to thesource storage device 112 once the application 108 reaches a known“good,” “stable” or “recoverable” state. In certain embodiments, such astate may be defined as when particular computing operations of theapplication 108 are complete to a point such that further operation,recovery and/or rolling back of the application 108 may occur, based onthe recorded data, without the loss of critical information or computingoperations needed for operation of the application 108. This point ofreferential integrity is generally referred to herein as a known goodstate of the application 108.

In certain embodiments, the data agent 236 instructs the quiescing ofthe application 108 through an application programming interface (API).For instance, the data agent 236 may send a command (e.g., FLRSNAP.FOO)to the application 108 that causes the application 108 to quiesce. Whenthe application 108 has placed itself in a known good state, theapplication 108 may send an acknowledgment to the data agent 236.

In certain embodiments, once the data management operations aresuspended, the I/O buffers in the data path of the application areflushed (and/or the writes in the queues are flushed), and the sourcelogs 244 are populated. For example, some or all of the pending datamanagement operations (e.g., as of the time of the suspension of theapplication) may be allowed to complete and/or percolate through thedata path. The filter driver 110 and/or data agent 236 then inserts alogical marker or tag in the source log file denoting that a“consistency point” or “consistency recovery point” has been reached. Insome embodiments, the consistency point indicates the time at which theapplication 108 is at a known good state. For instance, in certainembodiments, the data agent 236 instructs the filter driver 110 toinsert a consistency point entry into the source logs 244.

FIG. 4 illustrates consistency point entries as log entries 463 and 464in, respectively, the first and second log files 460, 462. As shown, theconsistency point entries are represented by “CRP” in the source logs244. In certain embodiments, once the consistency point is identifiedand inserted into the source log 244, the data agent 236 may instructthe application 108 to “restart” so as to resume normal data managementoperations from the application 108 to the source storage device 112.

Notwithstanding the foregoing, it will be understood that, in certainembodiments, although application 108 is quiesced, it need not actuallypause or suspend operation during the quiescent period. Rather, theapplication 108 may continue to operate substantially normally but mayinternally queue, or otherwise buffer, data management operationsintended for the source storage device 112. After the quiescent period,the buffered modification operations may be allowed to complete (i.e.,be sent to the source storage device 112).

In yet other embodiments, policies for the frequency of consistencypoint entries may be automatically generated. For instance, the dataagent 236 may be configured to quiesce the application 108 based on thestatus (e.g., capacity) of the source logs 244, the replication logs 352and/or the destination storage device 116. In yet other embodiments,quiescing of the application 108 may be performed based on an automaticreporting procedure. For instance, a module of the replication system400 may be configured to gather, receive and/or analyze informationassociated with a failure rate and/or health of applicable servers.Additional details of such status monitoring are provided in U.S. patentapplication Ser. No. 11/120,619, filed May 2, 2005, now published as US2006-0053261 A1, which is hereby incorporated herein by reference in itsentirety. For example, the frequency of consistency points may beselected or adjusted to mitigate risks detected in a storage network.

In certain embodiments, one or more log entries in the source logs 244are preferably associated with journal sequence numbers and/or timeinformation, such as, for example, assigned a time stamp indicative ofthe client system time with which the particular log entries areassociated. For instance, the time information may indicate the time atwhich: the log entry is written to the source log 244, the datamanagement operation is generated by the application 108, the datamodification operation is committed to disk or the like. In certainembodiments, not all the log entries are assigned a time stamp. Ratherparticular types of data, such as for example, consistency point markersand/or database commit entries, are assigned time stamps.

In certain embodiments of the invention, the data agent 236 coordinateswith the replication agent 356 to copy log files from the source logs244 to the replication logs 352. Such copying may be initiated based onany suitable factor, such as, for example, preset copying intervals,capacity thresholds reached in the source logs 244, time lapsed sincethe last copy operation, replication agent 356 requests for a copyoperation, and/or based on specific parameters or requirementsassociated with a particular application 108. For instance, certaindata-sensitive applications may be copied more frequently than otherapplications in order to reduce the amount of potential data loss due toa failure occurring between copy operations.

As further illustrated in FIG. 4, the replication logs 352 include afirst log file 466 and a second log file 468. In certain embodiments,each of these log files 466, 468 corresponds, respectively, to the firstlog file 460 and the second log file 462 of the source logs 244. Forinstance, data may be transferred between the replication log(s) 352 andthe source log(s) 244 such that the order in which the data was storedin the source log(s) 244 is preserved. In addition, the log files may berecreated in the replication log(s) 352 to reflect the organization ofsource logs 244. For example, the first log file 460 and the second logfile 462 in the source logs 244 may be transferred and recreated by thereplication agent 356 and/or the data agent 236. In other embodiments,however, data may be transferred and stored in a different order withoutpreserving source system correlations and/or may be rearranged on orduring transfer to or upon arrival in replication volumes 116A, 116B.

The illustrated destination system 104 further comprises an optionalpreference database 470 in communication with the replication agent 356.In certain embodiments, the preference database 470 includes storagepolicies or other preferences usable by the replication agent 356 inmanaging data. For instance, the stored preferences may indicate thedesired frequency at which the threads 358 should copy the data from thedestination logs 352 to the replication volumes 116A, 116B. Thepreference database 470 may also store path information for detailing towhich location(s) on the replication volume(s) 116A, 116B the data inthe replication log(s) 352 should be copied. In yet other embodiments,the preference database 470 may include storage policies that dictateparticular criteria for performing one or more data managementoperations on the replicated data.

With continued reference to FIG. 4, the replication module 114 furthercomprises one or more processes, such as a replication set or a logprocessing module 469 with a first thread 358A and a second thread 358B.In certain embodiments, as discussed above, the threads 358A, 358B areinstantiated by the replication agent 356 to transfer data from thefirst and second replication logs 466, 468 to the first replicationvolume 116A and/or the second replication volume 116B.

In certain embodiments, the threads 358A, 358B utilize time stamp orother temporal information that enables processing and/or replaying ofmodification operations. For example, based on time stamp information,the threads 358A, 358B may rearrange the replication data such that thedata is stored on the one or more replication volumes in the properorder (e.g., the order in which the data was intended to be written tothe source storage device 112). In such embodiments, the replicated datamay be subsequently retrieved, recalled or otherwise accessed orprocessed and may be used to accurately restore the state of theapplication 108 as it existed at a given point in time. In yet otherembodiments, other data management operations (e.g., searching, dataclassification) may be performed on the replicated data.

In certain embodiments, instructions for the storage operations are sentfrom the data agent 236 on the source system 102. For instance, theinstructions may be included in the log file entries copied from thesource system 102. In yet other embodiments, the storage operations arecoordinated by the replication agent 356 (e.g., according to storagepolices stored in the preference database 470) in combination with, orindependent of, the data agent 236. In yet other embodiments, policiesfor storage operations may be stored in another system managementcomponent (e.g., a storage manager module).

In certain embodiments, a snapshot is taken for each volume in whichdata is being replicated. For instance, with reference to FIG. 4, firstthread 358A is writing to the first replication volume 116A, and secondthread 358B is writing to the second replication volume 116B. In suchembodiments, when the first and second threads 358A, 358B arrive at aconsistency point log entry, a snapshot is taken of the replicated datain each replication volume 116A, 116B.

In certain preferred embodiments, when the snapshot is performed at aparticular consistency point, the time of the snapshot is advantageouslylogically associated with the time that the consistency point wasgenerated at the client system 102 (e.g., the client system time of theknown good state of the application 108). For instance, the time stampof the consistency point may be used to logically assign a “time” to thesnapshot of the replicated data. In such a process, the snapshot of thereplicated data then appears as if the snapshot was directly taken onthe data in the source system 102 at the time of the consistency point.Such a process allows for the snapshot data to be viewed as a directcopy of the production volume data for a particular application (e.g.,source storage device 112) at a certain point in time (e.g., the time ofa known good state of an application).

While certain embodiments of storage operations have been disclosed asbeing usable with the replication system 400 of FIG. 4, a wide varietyof other storage operations may also be performed on the replicationdata and/or in conjunction with consistency point information. Forexample, other copies of the replicated data may be performed, such as,but not limited to, creation, storage, retrieval, migration, deletion,auxiliary copies, incremental copies, differential copies, HierarchicalStorage Management (“HSM”) copies, archive copies, backup copies,Information Lifecycle Management (“ILM”) copies, other types of copiesand versions of electronic data or the like.

In certain embodiments, after appropriate storage operations areperformed on the replicated data, a message may be sent to other systemmanagement components (e.g., a snapshot manager and/or optional storagemanager) indicating that the replication process is complete up to thetime stamp associated with consistency point. At this point, thereplication agent 356 may instruct copy operations associated with thethreads 358A, 358B to resume.

FIG. 5 illustrates an exemplary embodiment of a data structure of a logentry 500 usable with the replication systems described herein. Incertain embodiments, the log entry 500 comprises information regardingmodifications to data and/or files on the source storage device 112 andmay include, for example, information regarding: which file wasmodified, the time of the modification, the type of modification, therelative data, a unique identification, combinations of the same or thelike. For exemplary purposes, the various fields of the log entry 500will be described with respect to a data write operation in thereplication system 400 of FIG. 4.

In certain embodiments, the log entry 500 is initially generated by thefilter driver 110 and is stored in the source log 244. For example, thelog entry 500 may comprise a data word having a plurality of fields. Asillustrated, the log entry 500 comprises a log entry number field 502, apath field 504, a time stamp field 506, an application type field 508, awrite type field 510, a size field 512, a checksum field 514, an offsetfield 516 and a payload field 522.

The log entry number field 502 may include information regarding theentry number assigned to the log entry 500 for system managementpurposes such that entries may be tracked and reordered relative to oneanother if necessary. For example, as mentioned herein, log entries maybe arranged in a temporally sequential manner based on the applicationwrite operation with which the particular log entry 500 is associated.In certain embodiments, log entry numbers or other information may berecycled over time once all the numbers in a particular range have beenused. In yet other embodiments, the log entry number field 502 may beconfigured to store other types of identification data for labeling thelog entry 500.

The path field 504 may include information regarding the file path onthe source storage device 112 with which the data write operation wasassociated. For example, a path of “C:\DIR\USER\” may indicate that thelog entry corresponds to an operation writing data to a folder or fileon the source storage device having the designated pathname. In certainembodiments, the path field 504 may include an absolute file pathname.In other embodiments, the path field 504 may include an abbreviatedpathname, an FID, and/or an inode (e.g., for UNIX-based systems).

Moreover, the path field 504 may include information relating to the logentry's replication volume destination, and thus may be useful inestablishing or confirming correlation or pairing information used bythe thread(s) 358A, 358B. For instance, in certain embodiments, the filepath of a particular log file may be hard-coded to one or moreparticular replication volume(s).

The time stamp field 506 may include information relating to the timewhen the subject data write occurred. In certain embodiments, the timestamp is advantageously associated with the time of the client computer230 on which the application 108 is executing. For instance, the filterdriver 110 may access the source system time when generating the logentry 500. In other embodiments, the time stamp may be provided by thefilter driver 110 and/or may be relative to the replication system time.

The application type field 508 may include information identifying theapplication type with which the log entry 500 is associated (e.g.,MICROSOFT OUTLOOK data, MICROSOFT SHAREPOINT data, ORACLE data, SQLdata, MICROSOFT WORD data, MICROSOFT INTERNET EXPLORER data or thelike).

The write type field 510 may include information regarding the categoryof write data involved with the log entry 500. For instance, the writetype may identify if the log entry 500 is associated with a databasemodification, a log write, a database commit command, a consistencypoint or the like. In certain embodiments, the information in the writetype field 510 is used to implement parallelism between multiple threadswhen performing data replication. For instance, a first thread (e.g.,thread 358A) may handle log write commands, and a second thread (e.g.,thread 358B) may handle commit database commands. In certainembodiments, the data stored in the write type field 510 may be used forprioritizing the processing of various log entries (e.g., processing bythe threads 358).

The size field 512 may include information relating to the size (e.g.,the number of bytes) of the data being modified by the data writeoperation. In yet other embodiments, the size field 512 may containinformation relating to the size of other or additional segments withinthe log entry 500, such as, for example, the size of the payload field522.

The checksum field 514 may include information relating to errorchecking to ensure, for example, that the log entry 500, when createdand subsequently transmitted, contains the expected number of bits andhas not been corrupted or otherwise impermissibly changed. For instance,the checksum field 514 may store data representing the arithmetic sum ofsome or all of the fields in the log entry 500.

The offset field 516 may include information relating to the locationwithin a file or portion of data that the data write is occurring. Forinstance, if the subject data write operation is associated withmodifying the twentieth through the thirtieth bytes of a file or pieceof data fifty bytes long, the offset field 516 may store a value oftwenty. In such embodiments, the information in the offset field 516 maybe used jointly with the information in the size field 512 to identifythe entire portion of a file being modified. For instance, in the aboveexample the size field 512 may store a value of eleven to indicate thelength of the modified section (i.e., twentieth through thirtiethbytes).

The payload field 522 may include information relating to the datawritten from the application 108 to the source storage device 112. Thisinformation generally represents the application data captured by thefilter driver 110 for replication and may include additional informationfor the ongoing operation or reconstitution of the application 108.

It will be understood that the illustrative filter driver log entry 500shown in FIG. 5 merely represents one possible embodiment of a log entrysuitable for use with embodiments of the invention and that otherembodiments may be used if desired. For example, in other embodiments,the log entry 500 may comprise more or fewer fields to accommodate therequirements of the particular replication or storage operation systeminvolved and/or to achieve certain data or management goals, such asconserving memory, increasing processing speed and increasing the amountof information in each log entry. For instance, in certain embodimentswherein the path determination for a particular log file or log entry isdynamic, the log entry 500 may not include the path field 504. In otherembodiments, the log entry 500 may include a priority field that may beused for prioritizing replication and/or data management operations ofdata associated with the log entry 500.

In other embodiments, the log entry 500 may concern a file attributechange rather than a data write operation. In such embodiments, thewrite type field 510 may identify the log entry 500 as being associatedwith a file attribute change. Furthermore, the log entry 500 may storeinformation regarding the new file attribute but would not requireoffset or size values to be stored in the size field 512 and/or theoffset field 516.

In yet other embodiments, as discussed in more detail below, the logentry 500 may not have a payload portion. Such embodiments cansignificantly reduce the size of the log files and/or increase systemperformance since copies of the actual data entries are not needed.Rather, information stored in the log entry 500 can be used by a filesystem driver (e.g., filter driver 110) to obtain a copy of the datafrom the source storage device 112, when need. Such information can beobtained from the path field 504, size field 512, offset field 516and/or other data identification information, such as inodes, FIDs orthe like.

FIG. 6 illustrates another embodiment of a replication system 600similar to the replication system 400 of FIG. 4. As shown, thereplication system 600 further includes a storage manager 680 thatcommunicates with the source system 102 and the replication system 104.In certain embodiments, the storage manager 680 is a software module orapplication that is configured to direct the performance of one or morestorage operations and, in particular, the replication of data from thesource system 102 to the replication system 104. In further embodiments,the storage manager 680 may perform one or more of the operations orfunctions described above with respect to the data agent 236 and/or thereplication agent 356. For instance, the storage manager 680 may directand/or coordinate the performance of one or more storage operations onthe replicated data (e.g., snapshots of the replicated data).

In certain embodiments, the storage manager 680 maintains an index 682,such as a cache, for storing information relating to: logicalrelationships and associations between components of the replicationsystem 600, user preferences, management tasks, and/or other usefuldata. For example, the storage manager 680 may use its index 682 totrack the location and timestamps of one or more snapshots of thereplicated data. In certain embodiments, the storage manager 680 maytrack logical associations between one or more media agents (not shown)and/or storage devices.

The storage manager 680 may also use its index 682 to track the statusof data management operations to be performed, storage patternsassociated with the system components such as media use, storage growth,network bandwidth, Service Level Agreement (“SLA”) compliance levels,data protection levels, storage policy information, storage criteriaassociated with user preferences, retention criteria, storage operationpreferences, and other storage-related information. The index 682 maytypically reside on the storage manager's hard disk and/or otherdatabase.

As shown in FIG. 6, the storage manager 680 further communicates with adatabase 684. In certain embodiments, the storage manager database 684comprises a memory for storing system management information relating tothe replication of data. For instance, the database 684 may beconfigured to store storage and/or restore policies, user preferences,the status or location of system components or data, combinations of thesame and the like. In yet other embodiments, the database 684 may beconfigured to store information described above with respect to theindex 682. In yet other embodiments, at least a portion of the index 682may be stored on the database 684.

Additional details of storage manager modules useful with embodiments ofthe replication systems described herein are described in U.S. Pat. No.7,389,311, issued Jun. 17, 2008, which is hereby incorporated herein byreference in its entirety.

FIG. 7 illustrates a simplified flowchart of an initialization process700 in accordance with certain embodiments of the invention. Inparticular, the initialization process 700 concerns certain preliminaryprocesses and acts for setting up a system for performing datareplication, as disclosed herein. For exemplary purposes, theinitialization process 700 will be described hereinafter with referenceto the components of the replication system 400 of FIG. 4.

The initialization process 700 begins with Block 705, wherein one ormore data agent(s) 236 are installed on the client computer 230. Incertain embodiments, the data agent 236 may be installed remotely fromother portions of the replication system 400 based on a particular needor to conform to certain directives or resident storage policies. Inother embodiments, the data agent 236 may be installed locally by asystem user as desired. For instance, installation of the data agent 236may include deployment and installation of object code files andsupporting software.

In certain embodiments, the data agent 236 may be installed for eachapplication 108 of interest, or one or more data agents 236 may beinstalled for a larger number of applications 108. Furthermore, incertain embodiments, an installation guide such as a wizard or otherprogram may recommend the appropriate number and type of data agents 236to install (which may be performed substantially automatically based onapplication and system configuration information).

At Block 710, the installed data agents 236 may perform certainauto-discovery routines in order to determine basic system andapplication information. In some embodiments, the auto-discoveryroutines may be considered part of the installation process. Forexample, the data agent 236 may begin the auto-discovery process byscanning and evaluating the folder and directory structure of the clientcomputer 230 to determine which folders are used by a particularapplication 108. In certain embodiments, such information allows thedata agent 236 to identify and locate files or other informationnecessary to replicate the current operating state of the application108 of interest.

In certain embodiments, the scanning and evaluation process may involvescanning multiple physical and/or logical volumes associated with thesource storage device 112 and/or within a given network or enterprise tolocate the data and system configuration information necessary for datareplication.

After the appropriate resources have been discovered and examined, thedata agent 236 may identify, arrange, coordinate and/or queue thenecessary data within various locations or instances of the application108 to establish a platform for proper data replication (Block 715). Incertain embodiments, this process may be a precursor for performing theinitial seeding or synchronization operation described above.

Next, at Block 720, the data agent 236 communicates with the replicationagent 356. For instance, the data agent 236 may transmit to thereplication agent 356 information regarding the replication of data. Thedata agent 236 may also request information from the replication agent356 and/or other network management components for any information thatmay bear on, or be related to, the correlation or mapping of networkstorage paths for replication data. For example, the data agent 236 mayconsult the preference database 470 of the destination system 104, thepreference database 465 of the source system 102 and/or a storagemanager component, for correlation or pairing information. Based on thisinformation, data paths may be identified for use by threads 358 whencopying data from the replication logs 352 to the replication volumes116A, 116B. In certain embodiments, one or more data paths may bedynamically coded or determined, such as, for example, based on one ormore storage policies and/or preferences.

At Block 730, the initialization process 700 includes installing andinitializing the filter drivers 110. In certain embodiments, suchinstallation and/or initialization is based at least in part oninformation obtained by the data agent 236 during the discovery orscanning process (Block 710). For example, in certain embodiments, oneor more filter drivers 110 may be installed by the data agent 236 in theI/O path of the application(s) 108.

FIG. 8 illustrates a simplified flowchart of an embodiment of a processof taking a consistency replication point in accordance with certainembodiments of the invention. In particular, the replication process 800involves the copying of data from a source system to a destinationsystem. Furthermore, in certain embodiments, the replication process 800is configured to be performed after completion of the initializationprocess 700 of FIG. 7. For exemplary purposes, the replication process800 will be described hereinafter with reference to the components ofthe replication system 400 of FIG. 4.

The replication process 800 begins with Block 805, wherein the filterdriver 110 populates the source log(s) 244 with data associated with theapplication 108, such as data identified by the data agent 236. Asdiscussed in more detail above, such data may relate to data or filemodification operations being passed from the application 108 to thesource storage device 112. In certain embodiments, the filter driver 110populates the source logs 244 in a temporally sequential manner suchthat operations and data are recorded in time descending (or ascending)order (e.g., first operation at the top and last operation at thebottom).

In certain embodiments, the data is populated in the source logs 244 ina format similar to the structure of the log entry 500 of FIG. 5. Inother embodiments, the data may be populated in other suitable formatsto satisfy the requirements of the particular replication system. Forinstance, the log file format may comprise a two- or multi-columnstructure, wherein the information in a first column may indicate thetype of data operation performed, and the log entry's position in thelog file indicates the order of the operation relative to otheroperations in the log file. The information in a second column mayindicate the payload data associated with the data operation indicatedby the first column.

After or concurrently with Block 805, the data agent 236 or other systemcomponent pauses or quiesces the application 108 (Block 810). Asdiscussed above, such quiescing causes the application 108 totemporarily suspend data modification operations to the source storagedevice 112 once the application 108 reaches a known good state.

Once new modification operations are suspended and the associated sourcelog 244 is populated based on the modification operations up to theknown good state, the data agent 236 or other replication systemcomponent inserts a logical marker or tag in the source log 244 (Block815). This “consistency point” denotes that the state of the data issuch that the application 108 may be recovered or that further stableoperation from that point going forward is ensured. Once the consistencypoint is identified and established, the data agent 236 may restart theapplication 108 such that data modification operations from theapplication 108 to the source storage device 112 resume.

As referenced by Block 820, the data agent 236 or other systemmanagement component coordinates the transfer of the data in the sourcelogs 244. In certain embodiments, the data agent 236 coordinates withthe replication agent 356 to copy data from the source logs 244 to thereplication log(s) 352. For instance, the replication agent 356 and/ordata agent 236 may open a network path or a communication socket betweenthe source log(s) 244 and the replication log(s) 352. The log entries ofthe source log(s) 244 may then be transferred as described above topopulate the replication log(s) 352. In certain embodiments, as thereplication log 352 is populated, the replication agent 356 may alsoobtain configuration information from the data agent 236 or other systemmanagement component such as, for example, a storage manager. Suchconfiguration information may identify aspects of the set of informationbeing transferred as well as identify pairing information thatcorrelates certain types of replication data with certain replicationvolumes or other storage destinations.

At Block 825, the replication process 800 includes instantiating one ormore threads 358 to begin the transfer of data from the replicationlog(s) 352 to certain replication volumes 116A, 116B. In certainembodiments, the replication agent 356 is configured to instantiate oneor more of the threads 358A, 358B. In certain embodiments, the threads358 are instantiated and/or particularized based on pairing orcorrelation information received from a management component and/orbased on certain system configuration information (e.g., availablereplication volumes), data path information, the type of information inthe transferred data set, combinations of the same and the like. Forexample, the replication agent 356 may instantiate one or more threads358 that correlate certain data types with certain data volumes and mayspecify primary and alternate data paths.

Once instantiated, the threads 358 process and/or traverse thereplication log(s) 352 until a consistency point is encountered (Block830). In certain embodiments, when reaching a consistency point, thethread 358 stops scanning the replication log 352 and notifies thereplication agent 356 that the thread 358 has reached the consistencypoint (Block 835).

In certain embodiments, once all active threads 358 associated withtraversing the replication logs 352 have notified the replication agent356 that a consistency point has been reached, the replication process800 moves to Block 840. At this point, the replicated data stored in thereplication volumes 116A, 116B preferably represents a known good stateof the application 108.

At Block 840, the replication agent 356 suspends further operation bythe threads 358. For instance, the replication agent 356 may suspenddata writes to the destination volumes 116A, 116B. At this point, thereplication process 800 proceeds with Block 845, wherein one or morestorage operations (e.g., snapshots) may be performed on the replicateddata, which are described in more detail above.

As discussed above, one of the advantages of the embodiments of the datareplication systems disclosed herein is that such systems are capable oftranslating information intercepted by a filter driver on a first(source) system into information that is suitable for replay (e.g.,replication) on a second (destination) system. In certain embodiments,however, the identification of files or directories in the source systemmay not be suitable for use with the directory structure of thedestination system.

For example, in UNIX-based systems, such as SOLARIS and LINUX, filesystem operations are generally identified as operations on “inodes” (or“vnodes”) such that files are referenced by a unique inode number and/orby a combination of one or more directory inode numbers and a shortname. Such systems often utilize file name or pathname translationalgorithms to implement a user-level hierarchical view of the filesystem.

Such usage of inodes and short names, however, is not conducive forreplaying data modification operations on a second system, such asoccurs in the data replication systems disclosed herein. That is, a pathhaving one or more inodes and/or short names does not provide adestination system with the appropriate information for performing thereplicated data modification operation.

On certain operating systems (e.g., SOLARIS 10, LINUX 2.6) pathnametranslation may sometimes be performed within the operating systemkernel by traversing backwards a directory name lookup cache (DNLC).Using such translation systems in the data replication environment,however, may yield concurrency issues if certain locking processes arenot performed. For instance, in order to ensure that other threads orprocesses do not rename one of the components of a file's absolute pathbetween the time that the thread computes the absolute path and the timethat a relevant log entry is emitted, the DNLC would need to be lockedagainst updates from other threads during that period of time. Havingthis central lock on the DNLC, however, may impose severe performancepenalties on the entire operating system.

FIG. 9 illustrates a block diagram of an exemplary embodiment of a datapath 900 usable to generate journal entries. Portions of the data path900 can be configured to more efficiently perform pathname translationin a data replication system. For example, in certain embodiments, thedata path 900 is advantageously configured to convert inode numbers(such as those used inside the kernel driver and/or associated virtualfile system handlers) of a source system into absolute file pathnames tobe used on one or more replication systems. In certain embodiments, allor a portion of the pathname translation is advantageously implementedin the application space external to the kernel space (e.g., in“userland”), thereby reducing potential loads on the source system.

As shown, the data path 900 comprises a filter driver 910. In certainembodiments, the filter driver 910 is configured to monitor datamanagement operations, such as data write operations or file attributemodification operations, associated with a computer applicationexecuting on a source computer. For instance, such operations maycomprise changes to data in a production level memory. Examples ofembodiments of filter drivers usable with the data path 900 aredescribed in more detail herein.

The filter driver 910 is further configured to populate a queue 912 withlog entries, or “raw” journal entries, related to detected datamodification operations from the application. In certain embodiments,the log entries generated by the filter driver 910 are each associatedwith an inode that identifies to which directory and/or file on thesource storage device the associated data modification was directed. Thequeue 912 is configured to store the log entries until they areprocessed by a driver thread (or process) 914. In certain embodiments,the queue 912 is implemented in volatile memory on the source system.

The queue 912 forwards the log entries to the driver thread 914. Incertain embodiments, the driver thread 914 polls the queue 912 fornewly-generated log entries by the filter 910. The driver thread 914subsequently stores the log entries in a buffer 916. In certainembodiments, the buffer 916 may be labeled a “raw” buffer in that it isconfigured to store “raw” log entries, which were generated by thefilter driver 910 and/or which do not yet have an absolute filepathname.

In certain embodiments, the buffer 916 is a memory-based queue forstoring the log entries until processed by a database thread (orprocess) 918. In certain embodiments, the buffer 916 advantageouslyfacilitates and/or expedites the unloading of raw records from expensivedriver memory to swappable application memory. For instance, the buffer916 may comprise an application level-buffer of a size betweenapproximately 40 megabytes and approximately 60 megabytes. In certainembodiments, the buffer 916 is advantageously implemented as a first-infirst-out buffer.

In certain embodiments, the database thread 918 is advantageouslycapable of performing inode-to-pathname translation for each of the logentries in the buffer 916. After performing the translation, thedatabase thread 918 may send the log entry (with the absolute filepathname instead of the inode entry) to a desired destination, such as areplication system, for further processing.

In certain embodiments, the database thread 918 is configured to accessa pathname database 920 to enable the thread 918 to perform pathnametranslation. The pathname database 920 advantageously stores informationthat associates one or more inodes or short names with an absolute filepathname. In yet other embodiments, the pathname database 920 maycomprise other means or data for performing pathname translation,including, but not limited to, a flat table, customized code,combinations of the same or the like.

In certain embodiments of the invention, accessing the pathname database920 introduces delay into the data path 900. For example, at certainpoints in the replication process, the filter driver 910 may generatelog entries at a quicker pace than the pathname translations beingperformed by the database thread 918. For instance, high activity disklookups in the database 920 for each log entry may require more timethan the generation of the log entries by the filter driver 910.

In such embodiments, the buffer 916 is advantageously capable ofadapting itself to the speed of the database thread 918. For example,when the lookups by the database thread 918 are relatively fast, thebuffer 916 does not introduce significant delay into the data flow(e.g., relatively no performance degradation due to the buffer 916).Thus, the buffer 916 may be advantageously sized to be relativelytransparent to the data stream (e.g., has a small footprint). However,when the database lookups begin to slow down, the buffer 916 is able tostore multiple log entries until the database thread 918 is able tocatch up.

Other mechanisms may be used to prevent user applications fromover-running the data path components (e.g., the queue 912, the buffer916, etc.). For example, in some cases user processes can generateinput/output so fast that such components overflow and starts swapping.In such a case, the filter driver 910 (or other appropriate component,such as the driver thread 914) may throttle the input/output byintroducing small delays into the input/output path. For example, thefilter driver 910 may lengthen the delays when an in-memory queuemaintained by the filter driver 910 approaches a preconfigured limit.Where the input/output throttling does not remedy the situation, andoverflow still occurs, the system may abort and reinitialize thereplication process.

In certain embodiments, the database lookups by the database thread 918may become so time intensive that the maximum storage capacity of thebuffer 916 is reached. In such embodiments, the buffer 916 is configuredto provide disk swapping functionality to avoid overflow of the buffer916, which may result in memory problems and/or aborting replication.For instance, as shown in FIG. 9, the buffer 916 may store excess logentries in a folder in memory 922. In certain embodiments, the memory922 may comprise a disk and/or may be located on the storage device ofthe source machine.

FIG. 10 illustrates an embodiment of a pathname database 920 of the datapath 900 of FIG. 9. In particular, the pathname database 920 may beadvantageously accessed by the database thread 918 when determining anabsolute file pathname for one or more log entries.

The illustrated pathname database 920 is configured forinode-to-pathname translation, such as for a UNIX-based system. Inparticular, the pathname database 920 includes three columns: adirectory inode (or parent inode) column 1022, a short name column 1024and an entry inode column 1026. In yet other embodiments, as describedin more detail below, the inode information in the database 920 can bereplaced with FIDs.

In certain embodiments, each inode in a UNIX-based system is recorded asan entry in the pathname database 920. For instance, FIG. 10 illustratesa system having four inodes, each having a single entry in the entryinode column 1026 and having a value of “1” through “4.” Thecorresponding short name column 1024 identifies the short name of thefile or folder associated with the particular inode. For instance, entryinode “4” identifies a folder or file with the short name of “user,”while entry inode “1” identifies a root directory. The directory inodecolumn 1022, or parent inode column, identifies the inode of the parentdirectory to the particular entry inode. For instance, entry inode “3,”which has a short name of “file,” is a child of the folder with an inodeof “2.”

As can be seen from the illustrated pathname database 920, when thedatabase thread 918 receives a log entry with a particular inode, thedatabase thread 918 is able to access the pathname database 920 andconstruct an absolute file pathname using the information stored thereinfor transmission to the destination system.

FIG. 11 illustrates an embodiment of a simplified pathname translationprocess 1100, such as may be performed by the database thread 918 ofFIG. 11 in conjunction with the pathname database 920 of FIG. 10. Forexample, the pathname translation process 1100 may be used to translatean inode to a pathname, such as an absolute file pathname to be used bya destination system in replicating data.

As shown, the translation process 1100 begins at Block 1105, wherein thedatabase thread 918 receives a log entry to be processed. For example,with reference to FIG. 9, the database thread 918 may retrieve the logentry from a buffer 916. In certain embodiments, the log entrypreferably represents a data modification operation associated with aparticular application on the source system.

At Block 1110, the database thread 918 identifies the inode associatedwith the particular operation represented by the log entry. Forinstance, the inode may represent a file or folder to which data is tobe written. In other embodiments, the inode in the log entry mayidentify a file name to be modified or other data or file modificationoperation.

At Block 1115, the database thread 918 accesses the pathname database920 to acquire information for translating the inode to an absolute filepathname. In particular, the database thread 918 searches the entryinode column 1026 for an entry that corresponds to the value of the logentry inode. Once the corresponding inode entry is found, the databasethread 918 determines (and stores) the associated short name from theshort name column 1024 (Block 1120).

The translation process then proceeds with Block 1125. If the subjectinode does not correspond to the root directory (“/”), the databasethread 918 identifies from the directory inode 1022 the inode of theparent directory (Block 1130). The database thread 918 then searches theentry inode column 1026 for the parent directory inode (Block 1135) andadds the short name associated with the parent directory inode to theabsolute file pathname (Block 1140).

The translation process 1100 then returns to Block 1125 to repeat thelookups and construction of the absolute file pathname until thedatabase thread 918 reaches the root directory. Once the database thread918 reaches the root directory, the database thread 918 stores the fullytranslated file pathname with the associated log entry (Block 1145), andthe translation process 1100 terminates.

For exemplary purposes, the translation process 1100 will be now bedescribed with reference to a data write command “vop_write (4, DATA)”and the values illustrated in the pathname database of FIG. 10. To beginthe translation process, the database thread 918 receives the log entryrepresenting the command “vop_write (4, DATA)” (Block 1105) whichcorresponds to writing “DATA” to inode “4” on the source system (Block1110).

The database thread 918 then accesses the pathname database 920 andsearches the entry inode column 1026 for a value of “4” (Block 1115).Upon finding “4” in the entry inode column 1026, the database thread 918determines from the short name column 1024 that the short namecorresponding to inode “4” is “user” (Block 1120).

Because inode “4” does not correspond to the root directory (Block1125), the database thread 918 identifies from the directory inodecolumn 1022 that the parent directory inode of inode “4” is inode “2”(Block 1130). The database thread 918 then returns to search the inodeentry column 1026 for the inode value of “2” (Block 1135), determinesthat the short name for inode “2” is “dir,” and adds “dir” to the filepathname (Block 1140).

Because inode “2” does not correspond to the root directory (Block1125), the database thread 918 identifies from the directory inodecolumn 1022 that the parent directory inode of inode “2” is inode “1”(Block 1130). The database thread 918 then searches the inode entrycolumn 1026 for the inode value of “1” (Block 1135) and determines thatthe inode “1” corresponds to the root directory (“/”) (Block 1140).

Now that the database thread 918 has encountered the root directory(Block 1125), the database thread 918 stores the translated filepathname (i.e., “/dir/user”) with the subject log entry, and thetranslation process 1100 terminates.

It will be understood that the translation process 1100 may differ inother embodiments of the invention in order to suit the needs of theparticular system(s) involved. For instance, the translation process1100 may be used to translate particular inodes into file pathnamesshorter than an absolute file pathname, such as for example a relativepathname. In yet other embodiments, the process can use FIDs in place ofinodes to construct absolute file names of files on the source system.

In certain embodiments, the three-column database 920 providessignificant advantages over a flat two-column table (e.g., with an inodecolumn and an absolute file pathname column). For instance, thethree-column database structure of the pathname database 920 may useless memory than the two-column table and/or expedite folder renameoperations. As an example, when a name of a folder is modified, thethree-column database structure allows for a single lookup andmodification (e.g., modifying the short name column 1024 entryassociated with the entry inode column 1026 entry of the subject inode),while the two-column table would require multiple lookups andmodifications corresponding to each entry having an absolute filepathname that includes the folder to be renamed.

As discussed above, in certain embodiments, the pathname database 920 ismaintained in userland (e.g., an application space external to thekernel space). In such embodiments, the pathname database 920 may beadvantageously managed and/or accessed by userland code withoutimpacting the resources of the operating system kernel or otherapplications.

In certain embodiments, the pathname database 920 may be initiallypopulated during an initialization period. For instance, a snapshot maybe taken to produce a static image of the file system of the sourcesystem. The pathname database 920 may then be populated based on thesnapshot. As subsequent changes are made to file names of the sourcesystem, corresponding changes are made in the pathname database 920 inorder to maintain synchronization. In yet other embodiments, asdiscussed in more detail below, the pathname database 920 can bepopulated based on scan of a live source file system.

In yet other embodiments, the pathname database 920 may be specific tothe files and/or folders of one or more particular applications. Forexample, the pathname database 920 may include inodes, short names andrelated information only for those inodes affected by a singleapplication (e.g., MICROSOFT EXCHANGE). In yet other embodiments,multiple pathname databases 920 may be used.

As can be appreciated in the translation systems and methods describedwith reference to FIGS. 9-11, it can be important for the file system onthe destination system to be in a synchronized state with the pathnamedatabase 920; otherwise the replication system can encounter a filesystem error (e.g., a file directory does not exist) when attempting toapply a journal entry on the destination system. In certain embodiments,this error can result in replication failure and require aresynchronization of the both the source and destination systems via theinitial seeding or synchronization process discussed above.

Moreover, as discussed, certain embodiments of the initialsynchronization process include performing an initial file system scanand populating the pathname database 920. As mentioned, this can beperformed by scanning a file system snapshot of the source system, whichobtains a static image of the file system. In particular, thereplication system can take a consistent snapshot in which, for theduration of the snapshot (e.g., the file system is flushed, frozen, andsnapped by file system driver), no namespace changing operations areallowed, such as renames of a directory, deletes, and/or creates. If thereplication system detects that some of these operations have occurredduring the synchronization process, the replication system may need todelete the snapshot and re-perform the scan. In certain circumstances,especially in active file systems, the replication system can gettrapped in a virtually infinite loop trying to take the snapshot overand over again due to the constantly changing files.

In yet other embodiments, the source file system may not support thetaking of consistent snapshots or may require additional drivers to beinstalled, thereby complicating the snapshot process. Moreover, takingsnapshots of a root file system can introduce even furthercomplications. For example, in some cases a root file system isallocated on a file system on which a snapshot cannot be taken, or onwhich it is difficult to take a snapshot. For instance, in Linux basedsystems the root file system is often located on a separate partition,outside of the Linux Volume Manager (LVM). Moreover, system directories(e.g., /etc., /var/tmp, /tmp) are often not sub-divided and aretherefore all located on the same root file system. In such cases, itcan be difficult to take a snapshot because modifications to thesedirectories occur on an on-going, continual basis.

Thus, certain embodiments of the invention provide systems and methodsfor producing a consistent image of a live source file system in apathname, or file name, translation database and/or on a destinationsystem without requiring a static image of the file system. Suchembodiments can advantageously allow for changes to the source filesystem to occur while other portions of the file system are beingscanned, thereby expediting the initial seeding of the file namedatabase and/or destination system and replaying of intercepted datachanges on the replication system.

For example, certain embodiments of the invention in a UNIX environmentutilize FIDs for performing snapless synchronization and replication ina CDR system. In certain embodiments, each FID comprises a sequence ofbetween eight and sixteen bytes that uniquely identifies a file orfolder in a file system.

In certain embodiments, the FIDs are introduced part of a UNIX kernelfor supporting stateless implementation of Network File System (NFS)version 3 or below. For example, the NFS 3 file system can access filesand directories via handles, in which the file system encodes allrelevant information that it needs to later translate the FID to thecorresponding file or directory inode.

In certain embodiments, NFS does not interpret contents of handles, butit uses the contents to directly refer to the files and directories ofinterest. For instance, these handles can contain FIDs, file/directoryinode numbers, generation numbers or the like. Moreover, file systemsthat are NFS-compatible (i.e., can be exported via NFS) can support theuse of FID scanning, as discussed in more detail below.

The use of FIDs can provide several advantages both during scanning andduring replication, such as for improving writes to the destinationsystem. For instance, systems can address subdirectories and carry onwith scanning even while the user makes changes to the file system,including such changes as renaming parent folders. Moreover, because thefile system translates FID information to locate files on a storagedevice (e.g., mapping of FIDs to vnodes), the FIDs can be used toidentify and access files or folders that are renamed or moved in a filesystem.

Another advantage is that generation IDs that are encoded into FIDs giveadditional robustness. For instance, if a file or directory is deletedand then recreated elsewhere during scanning, the file system may reusean inode number. However, file systems using FIDs generally incrementthe generation ID portion of the FID with each new file system object,thereby resulting in an absolute unique FID. Thus, FIDs are unique bothin space and in time, and using them can reduce the chance ofaccidentally confusing an old file system object with a recreated one.

FIG. 12 illustrates a block diagram of another embodiment of a pathnameor file name translation system 1200 that is configured to moreefficiently perform file name translation in a data replication system.For example, in certain embodiments, the translation system 1200 isadvantageously configured to associate FIDs, which are generally usedprimarily inside the kernel, with file short names and store theassociations in a database. The system 1200 can be further configured toconvert FIDs referencing files on a source system into absolute filenames to be used on one or more replication systems. This translationcan be advantageously implemented in the application space external tothe kernel space (e.g., userland), thereby reducing potential loads onthe source system.

As shown, the system 1200 includes the file system 1202 of the sourcecomputing system. In certain embodiments, the file system 1202 comprisesa UNIX environment implementing NFS. In other embodiments, the filesystem 1202 can comprise a NFS-compatible file system.

In communication with the file system 1202 is a scanning module 1204. Incertain embodiments, the scanning module 1204 is configured to scan alive file system (e.g., file system 1202), to build a database of FIDsand associated short names that reflect the structure of the file system1202, such as during an initial seeding or synchronization phase of datareplication. For instance, the scanning module 1204 can advantageouslypopulate the database without performing a snapshot of the source filesystem 1202.

In certain embodiments, the scanning module 1204 can comprise one ormore filter drivers, such as file system drivers that execute on acomputing device, such as the source computing device. In certainembodiments, the scanning module 1204 can comprise one or more dataagents 236. In yet other embodiments, the scanning module 1204 cancomprise a plurality of modules, either in software or hardware, thatare configured to perform the functions described herein.

In particular, the scanning module 1204 maintains a FID queue 1206 toassist with producing a consistent image of the live source file system1202. For instance, the FID queue 1206 can store a plurality of FIDsprocessed by the scanning module 1204 to populate a database. In certainembodiments, the queue 1206 comprises a first in-first out (FIFO) bufferor other like memory.

The system 1200 further comprises a database thread 1208 configured totranslate FIDs to absolute file names for replaying operations (e.g., asstored in a replication log file) on a destination system. For example,after performing the file name translation, the database thread 1208 maysend a log entry (with the absolute file name instead of the FID) to adesired destination, such as a replication system, for furtherprocessing.

In certain embodiments, the database thread 1208 is configured to accessa file name database 1210 to enable the thread 1208 to perform file nametranslation. The file name database 1210 advantageously storesinformation that associates one or more FIDs with short names anddirectory information. For instance, the file name database 1210 can besimilar to the pathname database 920 illustrated in FIGS. 9 and 10, withinode information of the pathname database 920 being replaced with FIDinformation. In yet other embodiments, the file name database 1210 maycomprise other means or data for performing file name translation,including, but not limited to, a flat table, customized code,combinations of the same or the like.

FIG. 13 illustrates a flowchart of an exemplary embodiment of a process1300 for scanning a live file system. In particular, the process 1300can be advantageously used to scan a source file system in a replicationenvironment, such as a CDR environment, without performing a snapshot onthe source file system. Such file system scanning can be less sensitiveto user changes during the scan and is able to interleave journalentries generated after the scan with a file name database (e.g.,database 1210) populated as part of the scan.

For example, the process 1300 may be performed on the file system 234 ofthe source system 102 of FIG. 2. For exemplary purposes, the process1300 will be described with reference to the components of the file nametranslation system 1200 of FIG. 12.

As shown, the process 1300 begins with Block 1305 by establishing anempty queue, such as queue 1206, for holding FIDs during scanning of thefile system 1202. At Block 1310, the process 1300 also creates an emptyfile name database 1210, as described above.

The scanning module 1204 then adds the FID of the source file system'sroot directory to the queue 1206 (Block 1315) and obtains a filedescriptor by opening the root directory in read-only (RO) mode (Block1320). The scanning module 1204 can obtain the file descriptor byissuing an open( ) call, for example. The file descriptor may comprisean integer or other appropriate identifier in userland, and can be usedas a file handler or file identifier for input/output. Behind the scenesin the kernel, the file descriptor number can be associated with thecorresponding file object. Thus, when a userland application writes somedata to a file descriptor, the kernel is aware of what object the datashould be written to. At this point, after obtaining the filedescriptor, the process 1300 begins a recursive procedure for steppingthrough each of the directories in the file system 1202 and populatingthe database 1210 with information usable to recreate a consistent imageof the file system 1202 on the destination system.

As shown, at Block 1325, the scanning module 1204 obtains the next FIDfrom the queue 1206. In the initial pass through the process 1300, theFID will generally be the root directory FID. At Block 1330, thescanning module 1204 asks the filter driver to associate, in the kernel,the appropriate previously obtained file descriptor with the currentFID. In certain embodiments, the scanning module 1204 invokes an ioctl() API (e.g., FLR_OPEN_BY_FID (fd, FID)) that receives both a pre-openfile descriptor and an FID. In response, a file system filter driverthen converts the FID to a file system vnode via a file-system-providedAPI and inserts the obtained vnode into the handler or file structurecorresponding to the passed file descriptor. Once this has completed,the application can then access the file or directory by making usualsystem calls and passing them the modified file descriptor.

At Block 1331, the scanning module 1204 scans the directorycorresponding to the FID using the associated file descriptor. Forexample, the scanning module 1204 steps into the directory associatedwith the current FID, such as through invoking the fchdir(fd) command,and reads each of the direct directory children, such as through theopendir(“.”) and readdir( ) commands. At Block 1335, for each detectedsubdirectory, the scanning module 1204 appends the FID associated withthe subdirectory to the end of the queue 1206 for further analysis.

At Block 1340, for each of the immediate directory children, thescanning module 1204 also populates the file name database 1210 with theFID and relative (short) name information. For instance, the scanningmodule 1204 may insert a row in the database 1210 that includes a parentdirectory FID, a short name of the file or folder and the entry's FID(see, e.g., FIG. 10).

Moving to Block 1345, the scanning module 1204 determines if there areadditional FIDs stored in the queue 1206. If so, the scanning module1204 returns to Block 1325 to obtain the next FID from the queue 1206and to step through the immediate children of the directory associatedwith that FID. In certain situations, with the continuously changingfile system and the possibility that the same directory is scanned morethan once, the scanning module 1204 can further resolve possiblestructural inconsistency problems between the scan list of directchildren and the contents of the file name database 1210 (Block 1346).This technique is described in further detail below (e.g., with respectto the process 1400 of FIG. 14).

When there are no additional FIDs stored in the queue 1206, the process1300 concludes and monitors log or journal entries for any changes tothe source file system directories (Block 1350).

As can be seen, with the process 1300, the scanning module 1204 does notaddress directories or files by absolute file names. Rather, thescanning module 1204 scans each of the file system directoriesindividually by addressing each directory by its unique FID, and bypopulating the database 1210 with the FIDs of children, along with theirrelative (short) names. UNIX systems typically do not allow directuserland access to file system objects using FIDs. Thus, the process1300 generally constructs a dummy file descriptor that is initiallyassociated with the root directory. The filter driver then locatesdesired file or directory objects (e.g., children files or directories)by their FID and associates those objects with the dummy filedescriptor. Thus, a userland application can then, e.g., use a “readdirectory” operation to obtain the list of children.

One of the benefits of the snapless scanning process 1300 of FIG. 13 isthat the file system 1202 can undergo changes by the user duringscanning without requiring a rescan of the entire file system. However,in certain circumstances, although the user's changes are intercepted bya file system driver and are appended to a change journal for furtherreplay, and although the scanning module 1204 is made less sensitive touser's changes by using the FID-driven scan, replication processesdisclosed herein can still encounter problems in the replicating phasewhen trying to replay collected journal entries on the destinationsystem and/or when performing database lookups.

For instance, in the case of snapshot-based scanning (e.g., FIGS. 9-11),the image on the destination system and the contents of the pathnamedatabase 920 are populated based on a point-in-time replica of theentire file system (i.e., the snapshot). Thus, all pending journalentries can be applied to the database 920 and/or replayed ondestination system following the initial scan in the order that theywere generated because, logically, the journal entries are generatedafter the snapshot.

However, in the case of snapless scanning (e.g., FIGS. 12 and 13), eachdirectory is scanned at a different time and, likely, during usermodifications to different portions of the file system. Thus, it becomesimportant to know if a particular folder or directory was scanned beforeor after a particular journal entry was generated. That is, if a journalentry associated with the contents of a folder is generated before thescanning of the folder, it may not be appropriate to apply the journalentry to the folder. Otherwise, there would be a risk of introducinginconsistencies into the file name database 1210, which can require therescanning of one or more directories in order to repopulate thedatabase 1210.

To address such risks of inconsistencies, in certain embodiments of theinvention, when scanning a particular directory during an FID-drivenscan, systems and methods can query the file system driver for itscurrent journal sequence number. The sequence number is then stored inthe file name database 1210 or other location along with theidentification of the current directory's children. When the system isto apply a stream of journal entries to the database 1210, the systemcan ignore all journal entries that were generated before the subjectdirectory was scanned, as identified by the stored journal sequencenumber.

FIG. 14 further illustrates a flowchart of an exemplary embodiment of ascan routine 1400 for obtaining both structural information from a filesystem (e.g., which FID represents a child of which parent directoryFID) and the time at which the particular relationship was observed. Incertain embodiments, this information is stored in the same database. Inparticular, the scan routine 1400 can be implemented as part of the scanprocess 1300 (e.g., at Blocks 1325 to 1340) to improve FID-drivenscanning and preserve consistency between the source file system imageand the file name database. For exemplary purposes, the process 1400will be described with reference to the components of the system 1200 ofFIG. 12.

The scanning module 1204 obtains the first directory FID from the queue1206 and associates a file descriptor with the FID, such as via a ioctl() call (e.g., a FLR_OPEN_BY_FID(fd, FID) command). For example, theseactions may be performed in the manner described above with respect toFIG. 13 (Blocks 1305 and 1310). At Block 1415, the routine 1400 stepsinto the directory of the current FID. For example, as discussed above,this can take place by the scanning module 1204 invoking the fchdir(fd)command.

At Block 1420, the scanning module 1204 obtains the current journalsequence number. In certain embodiments, the sequence number is assignedby a file system filter driver to each newly generated journal entry,being incremented with elementary changes made to the file system. Incertain embodiments, the scanning module 1204 obtains the sequencenumber from the filter driver and can be used as a measure of time andto advantageously coordinate the file system scan results with thestream of journal entries generated due to user's changes to file systemdata.

Upon obtaining the journal sequence number, the scanning module 1204begins monitoring the current directory for changes (Block 1425). Forinstance, the scanning module can invoke an ioctl( ) call that takes theFID of the current directory and initiates collecting statistics for thedirectory in the driver. In certain embodiments, collecting thestatistics comprises utilizing a counter of namespace changingoperations applicable to the current directory and intercepted by thedriver.

The scanning module 1204 then obtains the FID for each immediate childin the directory (Block 1430), as discussed in more detail above withrespect to Block 1335. After processing each of the immediate childrenin the current directory, the scanning module 1204 determines if therewere any structural changes to the directory during the scan (Block1435). For instance, the scanning module 1204 could issue an ioctl( )call that stops the collecting of statistics (e.g., by the driver) andreturns the number of namespace changing operations that happened sinceBlock 1425.

If there were structural changes to the directory during that time, theroutine 1400 assumes that the scan was not clean and repeats the scanprocess for the current directory by returning to Block 1425. On theother hand, if no changes are detected, the routine continues on withthe file scanning process. For example, the routine proceeds to populatethe file name database 1210 with the FIDs of direct children of thecurrent directory. This can occur in the fashion described above withrespect to the process 1300 of FIG. 13 (Block 1340). In certainembodiments, the process 1400 also includes storing the scan sequencenumber obtained at Block 1420 for the current directory. The routine inone configuration also stores the sequence number obtained at Block 1420in the file name database 1210. For example, the stored sequence numbercan then be used during replication to apply collected log entries, asdescribed below with respect to FIGS. 15A-15B.

In certain situations, with the continuously changing file system andthe possibility that the same directory is scanned more than once, thescanning module 1204 can further resolve possible structuralinconsistency problems between the scan list of direct children and thecontents of the file name database 1210. For instance, the scanningmodule 1204 can request a rescan of suspicious file system objects byappending their FIDs to the queue 1206 and/or by re-parenting to“null—fid” all database children of the current directory that are notidentified on the scan list.

In certain embodiments, for files that are on the scan list but are notidentified in the database 1210 as children of the current directory,the routine 1400, such as through the database thread 1208, can add rowscorresponding to the files in the database 1210. For subdirectories thatare on the scan list but are not identified in the database 1210 aschildren of the current directory, the routine 1400 can determined ifthe database already has an entry for the subdirectory. If so, and thesubdirectory is identified as a child of another directory, the databasethread 1208 can re-parent the subdirectory to the current directory andrequest a re-scan of the previous parent by obtaining its FID andappending it to the queue 1206. Otherwise, the database thread 1208 canadd a new row describing that the subdirectory is the child of thecurrent directory and can append the FID of the subdirectory to the FIDqueue 1206, thereby ensuring that before the scan completed, the childsubdirectory will be recursively scanned as well.

After the file name database 1210 is initially populated, certainembodiments of the invention are configured to detect file systemchanges on the source system and replicate such changes to the file namedatabase 1210 and, ultimately, to the destination system. FIGS. 15A and15B illustrate a flowchart of an exemplary replication process 1500 forinterleaving a stream of journal entries with the results of the livefile system scan in the database 1210, such as generated by the process1300 and/or routine 1400. For exemplary purposes, the process 1500 willbe described will be described with reference to the components of thefile name translation system 1200 of FIG. 12.

In general, the process 1500 includes obtaining and comparing sequencenumbers of journal entries with scan sequence numbers of respective FIDsin the database 1210. Based on this comparison, the process 1500determines whether or not to apply the journal entry to the database1210 and destination system, to discard the journal entry, or to triggeran additional FID scan.

The process 1500 begins at Block 1505 by obtaining the next journalentry and its associated sequence number from the file system driver,such as the filter driver 110, or a source log 244. The process 1500determines if the current journal entry is associated with a rename (ormove) operation of a file or subdirectory from one parent directory toanother (Block 1510). If not, the process 1500 then determines if thecurrent journal entry is associated with a create or remove operation ofa file or subdirectory in a parent directory (Block 1515). If not, theprocess 1500 returns to Block 1505 to obtain the next journal entry.

If it is determined at Block 1515 that the journal entry is associatedwith a create or remove operation, the scanning module 1204 and/or thedatabase thread 1208 look up the parent directory's FID in the database1210 to obtain the FID's scan sequence number, such as discussed withrespect to Block 1420 (Block 1520). If the sequence number of thejournal entry is less than or equal to the FID sequence number (Block1525), the process 1500 disregards the journal entry under theassumption that the journal entry was generated before the scanning ofthe corresponding portion of the file system (Block 1530). The process1500 then returns to Block 1505 to obtain the next journal entry.

At block 1534, the process 1500 updates the database 1210 to reflect orinclude the obtained journal entry. If the sequence number of thejournal entry is greater than the FID sequence number in the database1210, the database thread 1208 obtains from the database 1210 theabsolute file names of both the parent directory and the created/removedentity (Block 1535). With the information, the database thread 1208sends the journal entry for replay on the destination system (Block1540). The process 1500 then returns to Block 1515 to obtain the nextjournal entry.

If at Block 1510, the journal entry is associated with a rename (ormove) operation of a file or subdirectory from one parent directory toanother, the process 1500 moves to Block 1545 to obtain from thedatabase 1210 the FIDs of both the source and destination parentdirectories. If the journal entry sequence number is greater than thesequence number associated with the scan of the source directory (Block1550), the process 1500 then determines if the journal entry sequencenumber is also greater than the sequence number associated with the scanof the destination directory (Block 1555). If it is not, the databasethread 1208 removes the child from the source directory in the database1210 (Block 1560) and converts the journal entry from a rename operationto a remove operation (Block 1565). The database thread 1208 then sendsthe journal entry to the destination system to remove the child from thesource directory (Block 1570). The process 1500 then returns to Block1505.

If at Block 1555 the journal entry sequence number is determined to begreater than the sequence number associated with the scan of the sourceand destination directories, the database thread 1208 applies thejournal entry to the database 1210 (Block 1575). The database thread1208 further obtains from the database 1210 the absolute file names ofthe involved file system objects (Block 1580) and sends the journalentry to the destination system for replay (Block 1585). The process1500 then returns to Block 1505.

If at Block 1550 it is determined that the journal entry sequence numberis less than or equal to the sequence number associated with the scan ofthe source directory, the process 1500 further determines if the journalentry sequence number is greater than the sequence number associatedwith the scan of the destination directory (Block 1590). If so, theprocess 1500 recognizes that the FID of the object being moved is not inthe file name database 1210. That is, the source directory was scannedafter the rename was detected, and the destination directory was scannedbefore the rename was detected, indicating that the scanning module 1204missed the moved file system object. In this situation, the process 1500repeats the file system scan beginning with the FID of the object movedin the rename operation (Block 1592). The process 1500 then returns toBlock 1505 to obtain the next journal entry.

If at Block 1590 it is determined that that the journal entry sequencenumber is less than or equal to the sequence number associated with thescans of the source and destination directories, the process 1500disregards the journal entry (i.e., occurred before scans of both sourceand parent directories) (Block 1595) and returns to Block 1505 to obtainthe next journal entry.

In certain embodiments, the use of FIDs in file system scanning and/orcausing the associated filter driver to refer to affected file systemobjects by FIDs can advantageously provide for more efficient handlingof write operations. For instance, written data does not need to bejournaled from the file system driver to userland. Rather, the FID ofthe modified file and the offset/length of the modified regions can besent to the userland application for use in reading the data directlyfrom the file by opening the file with the FID and by merging themodified byte ranges.

This process can provide several benefits. For instance, not pipingwritten data from the driver to the userland application or journal canprovide significant improvements in performance. That is, copying datafirst in the driver's memory, then passing the data to the userlandapplication and storing the data in the disk cache can be quiteexpensive. By not journaling the actual data, but obtaining the datawhen needed directly from disk, system performance can be improvedseveral times.

In yet other embodiments, the use of FIDs in combination with a slightdelay in the actual transfer of data to the destination system can allowreplication systems to accumulate a list of changed byte ranges inmemory. This can provide further advantages in that the replicationsystem can analyze the changed bytes and optimize and/or improvereplication of data to the destination system.

For example, in certain embodiments, inventive systems and methods cancombine multiple write operations into a single write operation based onthe FIDs and byte ranges associated with operations by one or moreapplications. For example, the journal entry stream identifying the dataoperations intended for the source file system can be modified to referto FIDs instead of inode numbers and to journal the offset and/or lengthof overwritten byte ranges instead of actual data. This allows systemsand methods to obtain written data directly from disk, thereby achievinga significant improvement in performance.

For instance, in certain embodiments, the file system filter driverand/or data agent(s) monitoring data operations can write repetitivewrites to a single location. In yet other embodiments, the file systemdriver can combine modified adjacent byte ranges into a single writeoperation. In further embodiments, the file system driver can readnon-combinable byte ranges in the order of increasing file offsets,thereby obtaining better performance from the file system and duringsubsequent replication.

In certain embodiments, the file system driver and/or data agent(s) canimprove replication with respect to temporary files. The phrases“temporary file” or “temporary data” are broad terms and are used hereinin their ordinary sense and include, without limitation, data that iscreated by a program or application (e.g., editors and compilers) forsome transitory purpose, but deleted later, generally within a shortperiod of time.

For instance, in conventional replication systems, when an applicationcreates a temporary file, the new contents of the temporary file aregenerally replicated from the source to destination system. A brieftime, later the REMOVE command is replicated from the source system thatdeletes the transferred data on the destination system, mimicking themanner in which the file was created and removed by the application(s)on the source system.

By introducing the slight delay in the replication process, such a byaccumulated a number of journal entries, inventive systems and methodsmay encounter an error (e.g., a “no such file or directory” or “file notfound” error) from the file system when attempting to read the contentsof a temporary file when, within the delay period, the temporary datahas been removed from the source system. As a result, the replicationsystem does not send the temporary data across to the destinationsystem, as the file system is not able to locate the deleted file by theFID.

FIG. 16 illustrates a flowchart of an exemplary process 1600 including adelay period as discussed above for improving continuous datareplication. In particular, the process 1600 addresses the analysis ofmultiple (e.g., two) data operations received from one or moreapplications during an introduced delay period; however, it will beappreciated that other embodiments of the invention can introduce longerdelays that capture additional data operations for analysis. Forexemplary purposes, the process 1600 will be described with reference tothe components of the replication system 400 of FIG. 4 utilizing anFID-driven replication procedure.

As shown at Block 1605, the process 1600 begins by receiving from thefilter driver 110 a first journal entry related to a modified file. Incertain embodiments, the file system filter driver 110 intercepts orotherwise accesses a data modification operation sent by theapplication(s) 108. At Block 1610, the data agent 236 identifies the FIDof the file to be modified on the source system 102 and the offset andlength of the modified portions of the file. In certain embodiments, thefilter driver 110 advantageously does not store or otherwise retain acopy of the actual data to be modified for each such data modificationoperation.

Also, instead of immediately transmitting to the destination system 104the logs and data associated with replaying the data operation on thedestination system 104, the process 1600 introduces a delay in thereplication of data. In certain embodiments, this delay is betweenapproximately three and four seconds, in other embodiments, the delaycan be of a shorter or longer duration.

Due to the delay, the data agent 236 receives at least a second journalentry before the data associated with the first journal entry isaccessed (Block 1615). At Block 1620, based on the second journal entry,the data agent 236 identifies the FID of the file to be modified on thesource system 102 and the offset and length of the modified portions ofthe file. Based on the data location information received from both thefirst and second journal entries, the data agent 236 determines if thedata modification operations from the two journal entries are writeoperations for the same data (Block 1625). If so, the filter driver 110processes only the later data write operation associated with the secondjournal entry and accesses the modified data portions on disk fortransmission to the destination system (Block 1630). The earlier dataoperation of the first journal entry is ignored as being out-of-date.

However, if the data operations are for different file regions, the dataagent 236 determines if the operations concern writes to adjacent byteranges that can be combined (Block 1635). For example, the data agent236 can determine if the distance between the two byte ranges is largerthan a predetermined threshold. For instance, the threshold can be basedon the size of overhead (e.g., a header) associated with journalentries. In certain embodiments, the threshold distance is 200 bytes. Inyet other embodiments, the distance can be larger (e.g., 1 KB) orshorter and/or dynamically adjusted.

If the distance between the two byte ranges is less than the threshold,the process 1600 combines the separate write operations of the first andsecond journal entries into a single journal entry having a singlewrite. In this case, the single write operation is replayed on thedestination system 104 with both byte ranges being replicated (Block1640).

If the byte ranges are sufficiently separated, the data operations fromthe two journal entries cannot be combined, and the process 1600 handlesthe journal entries separately (Block 1645). That is, the data agent 236accesses each of the modified portions of the file(s) based on theinformation in the two journal entries. If either of the data accessrequests results in a particular type of file system error, such as a“no such file or directory” or “file not found” error (Block 1650), theprocess 1600 discards the journal entry associated with the request(Block 1655). For instance, in certain embodiments, due to theintroduced delay, by the time the data is requested, the data may havealready been deleted, moved or removed, such as is the case withtemporary files.

Finally, if no error is received when trying to access the data, theprocess 1600 transfers the modified portions pertaining to each journalentry for replay and replication on the replication system 104. Incertain embodiments, the transfer and/or replay of the journal entriescan be performed in order of increasing file offsets, especially withjournal entries associated with the same FID.

Although the process 1600 is described with reference to particulararrangements, it will be understood that other embodiments of theinvention may have more or fewer blocks that those described above. Forinstance, the data location information extracted from the secondjournal entry can be further compared with data location information ofa third journal entry or additional journal entries, Thus, as can beseen, the process 1600 can be repeated for each subsequent journal entrycaptured by the filter driver 110.

Embodiments of the invention have been described herein with referenceto UNIX file systems and can include LINUX, XFS, Veritas, EXT3 filesystems and the like.

In certain embodiments of the invention, data replication systems andmethods may be used in a modular storage management system, embodimentsof which are described in more detail in U.S. Pat. No. 7,035,880, issuedApr. 5, 2006, which is hereby incorporated herein by reference in itsentirety. For example, the data replication system may be part of astorage operation cell that includes combinations of hardware andsoftware components directed to performing storage operations onelectronic data. Exemplary storage operation cells usable withembodiments of the invention include CommCells as embodied in the QNetstorage management system and the QiNetix storage management system byCommVault Systems, Inc. (Oceanport, N.J.), and as further described inU.S. Pat. No. 7,454,569, issued Nov. 18, 2008, which is herebyincorporated herein by reference in its entirety.

Systems and modules described herein may comprise software, firmware,hardware, or any combination(s) of software, firmware, or hardwaresuitable for the purposes described herein. Software and other modulesmay reside on servers, workstations, personal computers, computerizedtablets, PDAs, and other devices suitable for the purposes describedherein. Software and other modules may be accessible via local memory,via a network, via a browser, or via other means suitable for thepurposes described herein. Data structures described herein may comprisecomputer files, variables, programming arrays, programming structures,or any electronic information storage schemes or methods, or anycombinations thereof, suitable for the purposes described herein. Userinterface elements described herein may comprise elements from graphicaluser interfaces, command line interfaces, and other interfaces suitablefor the purposes described herein.

Embodiments of the invention are also described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products. It will be understood that eachblock of the flowchart illustrations and/or block diagrams, andcombinations of blocks in the flowchart illustrations and/or blockdiagrams, may be implemented by computer program instructions. Thesecomputer program instructions may be provided to a processor of ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer orother programmable data processing apparatus, create means forimplementing the acts specified in the flowchart and/or block diagramblock or blocks.

These computer program instructions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to operate in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meanswhich implement the acts specified in the flowchart and/or block diagramblock or blocks. The computer program instructions may also be loadedonto a computer or other programmable data processing apparatus to causea series of operations to be performed on the computer or otherprogrammable apparatus to produce a computer implemented process suchthat the instructions which execute on the computer or otherprogrammable apparatus provide steps for implementing the acts specifiedin the flowchart and/or block diagram block or blocks.

In addition, methods and functions described herein are not limited toany particular sequence, and the acts or blocks relating thereto can beperformed in other sequences that are appropriate. For example,described acts or blocks may be performed in an order other than thatspecifically disclosed, or multiple acts or blocks may be combined in asingle act or block.

While certain embodiments of the inventions have been described, theseembodiments have been presented by way of example only, and are notintended to limit the scope of the disclosure. Indeed, the novel methodsand systems described herein may be embodied in a variety of otherforms; furthermore, various omissions, substitutions and changes in theform of the methods and systems described herein may be made withoutdeparting from the spirit of the disclosure. The accompanying claims andtheir equivalents are intended to cover such forms or modifications aswould fall within the scope and spirit of the disclosure.

What is claimed is:
 1. A method for identifying data to be copied in adata replication system, the method comprising: using a computingdevice, adding a first file identifier descriptor (FID) of a firstdirectory on a live source file system to a queue, the first FID beingone of a plurality of unique identifiers corresponding to a plurality ofdirectories and files on the source file system; storing a currentjournal sequence number from a file system filter driver identifying afirst time; following said storing, accessing a current directory of theplurality of directories on the source file system that corresponds to anext FID stored in the queue; obtaining additional FIDs for eachimmediate child directory and immediate child file in the currentdirectory; and in response to determining that no changes have been madeto the current directory since the first time: populating a file namedatabase with the additional FIDs of each immediate child directory andimmediate child file in the current directory; adding the additionalFIDs of each immediate child directory of the current directory to thequeue; and removing the next FID from the queue.
 2. The method of claim1, wherein: in response to determining that changes have been made tothe first directory since the first time, said storing, said accessingand said obtaining the additional FIDs are repeated.
 3. The method ofclaim 1, wherein the first FID obtained by a scanning module executingon a computing device.
 4. The method of claim 1, wherein the first FIDis obtained without performing a snapshot on the source file system. 5.The method of claim 1, wherein the first directory is the currentdirectory.
 6. The method of claim 1, wherein said storing, saidaccessing and said obtaining the additional FIDs are repeated for eachFID stored in the queue.
 7. The method of claim 1, wherein saidpopulating the file name database comprises for each immediate childdirectory and immediate child file in the current directory, storing inthe file name database: the additional FID for the immediate childdirectory or immediate child file; a corresponding short name for theimmediate child directory or immediate child file; and the next FID as aparent directory of the immediate child directory or immediate childfile.
 8. The method of claim 1, wherein said changes comprise namespacechanges to the current directory.
 9. The method of claim 1, wherein thefirst directory comprises a root directory of the live source filesystem.
 10. The method of claim 1, further comprising monitoring atleast one data management operation directed to first data stored in thesource file system.
 11. The method of claim 10, further comprisingreplaying the at least one data management operation on replication datastored on a destination file system.
 12. The method of claim 11, furthercomprising: constructing, from information populated in the file namedatabase, an absolute file name that corresponds to a location of thefirst data on the source file system; and transmitting the absolute filename to the destination file system to direct said replaying of the atleast one data management operation.
 13. A system for preparing data forreplication from a source computing device in a network, the systemcomprising: one or more memory devices containing: a queue including aplurality of file identifier descriptors (FIDs) each comprising a uniqueidentifier that corresponds to one of a plurality of directories andfiles on a source file system; and a database comprising file name datathat associates each of the plurality of FIDs with a short name and aparent FID; a computing system comprising one or more computing devicescomprising computer hardware, the computing system configured to: scanthe source file system while in a live state and to populate the queuewith the plurality of FIDs; access a current directory of the pluralityof directories on the source file system that corresponds to a next FIDin the queue; and obtain additional FIDs for each immediate childdirectory and immediate child file in the current directory; andpopulate the database with the file name data based on said scan of thesource file system in the live state.
 14. The system of claim 13,further comprising: at least one database thread configured to receive adata entry identifying a data management operation associated with atleast one of the plurality of directories and files on the source filesystem and to construct from the FID associated with the at least onedirectory or file an absolute file name for transmission to adestination system along with a copy of the data management operationfor replaying on the destination system.
 15. The system of claim 14,further comprising a filter driver situated between the source filesystem and at least one application configured to request the datamanagement operation.
 16. The system of claim 15, wherein the filterdriver is further configured to assign journal sequence numbers to eachjournal entry associated with a requested change to the source filesystem.
 17. The system of claim 16, wherein the computing system isfurther configured to receive a current journal sequence number from thefilter driver prior to accessing the current directory.
 18. The systemof claim 17, wherein the computing system is further configured torepeat said accessing and obtaining in response to detecting changes tothe current directory following a time of the current journal sequencenumber but prior to said obtaining.
 19. The system of claim 13, whereinthe computing system is further configured to: populate the file namedatabase with the additional FIDs of each immediate child directory andimmediate child file in the current directory; and add the additionalFIDs of each immediate child directory of the current directory to thequeue.
 20. A system for identifying data to be copied in a datareplication system, the system comprising: one or more computing devicescomprising computer hardware and configured to: add a first fileidentifier descriptor (FID) of a first directory on a live source filesystem to a queue, the first FID being one of a plurality of uniqueidentifiers corresponding to a plurality of directories and files on thesource file system; store a current journal sequence number from a filesystem filter driver identifying a first time; following said storing,access a current directory of the plurality of directories on the sourcefile system that corresponds to a next FID stored in the queue; obtainadditional FIDs for each immediate child directory and immediate childfile in the current directory; and in response to determining that nochanges have been made to the current directory since the first time:populate a file name database with the additional FIDs of each immediatechild directory and immediate child file in the current directory; addthe additional FIDs of each immediate child directory of the currentdirectory to the queue; and remove the next FID from the queue.